You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'm working on a script using GitPython to loop over lots of Git repos, pull remote changes, etc...
At some point, I noticed that doing repo.is_dirty() was opening some files which never got closed until the process exits, which in turns causes the tool to crash with "too many open files" after iterating over enough repos:
importosimportgit# Put here whatever Git repos you might have in the current folder# Listing more than one makes the problem more visiblerepos= ["ipset", "libhtp"]
raw_input("Check open files with `lsof -p %s | grep %s`"% (os.getpid(),
os.getcwd()))
fornameinrepos:
repo=git.Repo(name)
repo.is_dirty()
delreporaw_input("Check open files again") # files are still open
I tried digging deeper down the GitPython stack, but couldn't find the actual cause.
In case that's helpful, below is the same as the previous snippet, but using directly the lowest-level gitdb objects I could find to open the files:
importosimportgitfromgitdb.utilimporthex_to_bin# Put here whatever Git repos you might have in the current folder# Listing more than one makes the problem more visiblerepos= ["ipset", "libhtp"]
raw_input("Check open files with `lsof -p %s | grep %s`"% (os.getpid(),
os.getcwd()))
raw_input("Check open files again") # files are still openfornameinrepos:
repo=git.Repo(name)
sha=hex_to_bin("71acab3ca115b9ec200e440188181f6878e26f08")
fordbinrepo.odb._dbs:
try:
foritemindb._entities: # db._entities opens *.pack filesitem[2](sha) # opens *.idx filesexceptAttributeErrorase:
# Not all DBs have this attributepassdelreporaw_input("Check open files again") # files are still open
But that's just where the files are open (well, in fact they are open even lower down but I didn't manage to go deeper), not where they are supposed to be closed.
Thanks for reporting this.
Probably this is related to the smmap library, which opens memory mappings of handled files. Apparently, there is a chance of not closing them after being done with the access.
I suffered from similar problems in windows tests (where open handles prevented the test-cases to clean themselves up properly), and this definitely should be looked into.
Until then, you might use the GitCmdDb Backend for your repository, which uses a different implementation.
I think, the lsof checks will have to go into the test-suite too, in order to protect against regression.
It will be a lesson for me, as I simply didn't verify I indeed close the handles (or that there are no dependency-cycles preventing objects to die).
Especially with something as vital (and limited) as system resources, I shouldn't leave anything to chance.
Currently I have no ETA on this, but I am planning to redirect development time to git-python once again.
In the case of this bug report, path_or_fd is a path, so the fd gets opened. At this point, it is file descriptor 3.
Then later on, in the same function, I can see:
finally:
ifisinstance(path_or_fd, basestring):
os.close(fd)
#END only close it if we opened it
And here, the file descriptor 3 is closed. So everything is good, right?
No, because in between, there is this:
self._mf=mmap(fd, actual_size, **kwargs)
This maps the content of the file pointed by file descriptor 3 in memory... and it also opens the file again, as file descriptor 4!
It is this one which never gets closed.
Now let's consider the following:
importmmapimportosraw_input("Check open files with `lsof -p %s | grep %s`"% (os.getpid(),
os.getcwd()))
f=open("hello", "r+b")
raw_input("Check open files again (opened 'hello' as fd %s)"%f.fileno())
mf=mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)
raw_input("Check open files again (mmaped fd %s)"%f.fileno())
f.close()
raw_input("Check open files again (closed 'hello')")
mf.close()
raw_input("Check open files again (closed the mmap)")
This reproduces exactly the same behaviour: a file is opened as fd 3, when mmap-ing its content we also open the same file as fd 4, then we close the fd 3, and finally closing the map will also close fd 4.
So what is missing from smmap is to close the memory map when done with it.
Thank you for making the effort of pointing the course of the issue out so clearly.
As this affects all git-python installation, I should definitely be on it with high priority !
In any way, the memory map needs to remain open until the last user of the respective memory region fades away, and apparently this doesn't happen, maybe because someone keeps it open by keeping a reference to it - this at least would be my guess.
AFAIK, when the mmap instance goes out of scope, it will close automatically, maybe I am relying on this. Also, smmap has a known issue with reference counting in some python versions apparently, which may also lead to the issue we see here.
As this affects all git-python installation, I should definitely be on it with high priority !
I'd certainly appreciate it. :)
Especially since it's not even possible to manually keep track of those files (by monkey-patching Python's open() and file()) and close them manually: the mmap module is written in C and doesn't use either open() or file().
In any way, the memory map needs to remain open until the last user of the respective memory region fades away,
I tried spending some time yesterday figuring where this should be, but that's just too much for me.
and apparently this doesn't happen, maybe because someone keeps it open by keeping a reference to it - this at least would be my guess.
Well, I tried destroying the repo object, but that didn't help. If a reference is kept on it, I really wonder where. :-/
Also, smmap has a known issue with reference counting in some python versions apparently, which may also lead to the issue we see here.
So I can confirm the issue on RHEL 6 (Python 2.6.6) and Fedora 16 (Python 2.7.2)
And mscherer above confirmed it on Fedora 17, which has Python 2.7.3.
So I'm not sure that's related to a Python version, or else we are extremely unlucky. :D
When I implemented the reference counting, I decided to try using Python's internal one, with sys.getrefcount(). However, I realized that I had to put some magic numbers into the code to account for references to my object that I didn't really have on my radar. In that moment, I might have opened the gates for plenty of bugs.
Or I have a cyclic relation somewhere, even though I was very conscious about strong references and about who keeps pointers to whom.
Anyway, its buggy, and a real problem as GitPython uses this engine heavily under the hood, and by default.
Wow, exactly one year ago I said I had no time, and sadly this still didn't change. However, next year I could possibly get to it, there are many things I want to improve and do with git-python.
That sounds great.
I've managed to make a workaround for the function that caused the "Too many open files" error using git commands.
Here's a simplified version of the code that caused the error:
import git
def last_commit(repo, rev, path):
commit = repo.iter_commits(rev, path, max_count=1)
return commit.next()
repo = git.Repo('/path/to/repo')
for i in range(1000):
print last_commit(repo, 'master', '/path/to/some/dir/inside/repo')
After about 500 iterations it crashes with
OSError: [Errno 24] Too many open files
Is there any way of getting the lastest commit from a directory or a file on a certain branch or tag besides using iter_commits ? I'd rather use objects instead of parsing raw git output and making my own dictionary with the values.
@virgiliu: sure. I just hope @Byron doesn't mind that we're using his bug tracker to discuss like this. :)
So, in one of my projects I'm using the following solution. I haven't pushed it yet (mostly because I'm not entirely satisfied with it) so I can't just give you a link to the code. You can assume that the code below is MIT-licensed.
So first, I have the following decorator:
classsubprocessed(object):
"""Decorator to run a function in a subprocess We can't use GitPython directly because it leaks memory and file descriptors: https://github.com/gitpython-developers/GitPython/issues/60 Until it is fixed, we have to use multiple processes :( """def__init__(self, func):
self.func=funcdef__get__(self, instance, *args, **kwargs):
self.instance=instancereturnselfdef__call__(self, *args):
frommultiprocessingimportProcess, Queueq=Queue()
defwrapper(queue, func, *args):
queue.put(func(*args))
p=Process(target=wrapper,
args=(q, self.func, self.instance) +args)
p.start()
p.join()
returnq.get()
Then, you just need to decorate a method with it, and it gets run as a subprocess!
In my case, I have a class Foobar which does stuff on various Git repositories:
classFoobar(object):
def__init__(self):
self.git_rooturl="git://git.example.com"self.workdir="/path/to/where/the/repo/will/be/cloned"defdo_stuff_on_repo(self, reponame):
curdir=os.getcwd()
# Clone the modulerepo=git.Repo.clone_from("%s/%s"% (self.git_rooturl, reponame),
os.path.join(self.workdir, reponame))
os.chdir(repo.working_tree_dir)
# Do your stuff hereresult= ...
os.chdir(curdir)
shutil.rmtree(repo.working_tree_dir)
delreporeturnresult
Unfortunately the do_stuff_on_repo method goes through the wrong code path and I call it repeatedly (even on different repositories), so it leaks memory and fds.
All I changed is that I decorated it, and now each call to the function will be run in its own process, so the leaking memory and fds get cleared when the process exits.
The only tricky part which is not handled above is what to do in case of an error: if do_some_stuff_on_repo raises an exception for example, then subprocess just exits, without affecting the main process where your application is running.
How to handle that properly is left as an exercise to the reader. (because that's the part I'm not very happy with :P)
The issue presented by @virgiliu seems to be fixed now (with the latest version of smmap 0.9.0), but by artificially constraining the file limit to 64, I can still see that pipes are leaked.
For some reason, repository instances are never freed, and I will have to figure out why that would happen.
As __del__() is not reliably called in python 3, the tests leak heavily as no repository it ever used gets cleaned up, leaving plenty of the processes shown above. Each of them comes with three pipes, quickly using up the processes resources.
For example, it's possible to run all tests with ulimit -n 64 in python 2, but needs ulimit -n 140 in python 3.
It turned out I forgot to add a tear-down super class call, which was responsible for most of the test-leakage.
On the plus side, travis now checks for this as well.
An improved leak-test was added, which clearly shows that repositories dispose of their file handles correctly.
At the same time, look at the number of open files with lsof | grep ".git". I see that the number of open files increases by about 25. The only way to decrease the number of open files is to close the python session.
(I should point out that I am running on Mac OSX, with a mostly MacPorts python installation.)
Activity
Byron commentedon May 17, 2012
Thanks for reporting this.
Probably this is related to the smmap library, which opens memory mappings of handled files. Apparently, there is a chance of not closing them after being done with the access.
I suffered from similar problems in windows tests (where open handles prevented the test-cases to clean themselves up properly), and this definitely should be looked into.
Until then, you might use the GitCmdDb Backend for your repository, which uses a different implementation.
mscherer commentedon May 17, 2012
I can confirm the issue on fedora 17, there is *.pack and *.idx, open as file and memory mapped.
Byron commentedon May 17, 2012
I think, the lsof checks will have to go into the test-suite too, in order to protect against regression.
It will be a lesson for me, as I simply didn't verify I indeed close the handles (or that there are no dependency-cycles preventing objects to die).
Especially with something as vital (and limited) as system resources, I shouldn't leave anything to chance.
Currently I have no ETA on this, but I am planning to redirect development time to git-python once again.
bochecha commentedon May 17, 2012
So, learning my way through the code with
pdb, I found where the file is actually opened.In
smmap/util.py, in the constructor of the classMapRegion, there's the following:In the case of this bug report,
path_or_fdis a path, so thefdgets opened. At this point, it is file descriptor 3.Then later on, in the same function, I can see:
And here, the file descriptor 3 is closed. So everything is good, right?
No, because in between, there is this:
This maps the content of the file pointed by file descriptor 3 in memory... and it also opens the file again, as file descriptor 4!
It is this one which never gets closed.
Now let's consider the following:
This reproduces exactly the same behaviour: a file is opened as fd 3, when mmap-ing its content we also open the same file as fd 4, then we close the fd 3, and finally closing the map will also close fd 4.
So what is missing from
smmapis to close the memory map when done with it.Byron commentedon May 18, 2012
Thank you for making the effort of pointing the course of the issue out so clearly.
As this affects all git-python installation, I should definitely be on it with high priority !
In any way, the memory map needs to remain open until the last user of the respective memory region fades away, and apparently this doesn't happen, maybe because someone keeps it open by keeping a reference to it - this at least would be my guess.
AFAIK, when the mmap instance goes out of scope, it will close automatically, maybe I am relying on this. Also, smmap has a known issue with reference counting in some python versions apparently, which may also lead to the issue we see here.
bochecha commentedon May 18, 2012
I'd certainly appreciate it. :)
Especially since it's not even possible to manually keep track of those files (by monkey-patching Python's
open()andfile()) and close them manually: themmapmodule is written in C and doesn't use eitheropen()orfile().I tried spending some time yesterday figuring where this should be, but that's just too much for me.
Well, I tried destroying the repo object, but that didn't help. If a reference is kept on it, I really wonder where. :-/
So I can confirm the issue on RHEL 6 (Python 2.6.6) and Fedora 16 (Python 2.7.2)
And mscherer above confirmed it on Fedora 17, which has Python 2.7.3.
So I'm not sure that's related to a Python version, or else we are extremely unlucky. :D
Byron commentedon Jun 7, 2012
When I implemented the reference counting, I decided to try using Python's internal one, with sys.getrefcount(). However, I realized that I had to put some magic numbers into the code to account for references to my object that I didn't really have on my radar. In that moment, I might have opened the gates for plenty of bugs.
Or I have a cyclic relation somewhere, even though I was very conscious about strong references and about who keeps pointers to whom.
Anyway, its buggy, and a real problem as GitPython uses this engine heavily under the hood, and by default.
bochecha commentedon Jul 9, 2012
Any news on this issue?
Byron commentedon Jul 11, 2012
No, sorry. Also I cannot currently assign any time to the project.
virgiliu commentedon Aug 28, 2013
Any news? I'm getting the same error on Python 2.6.6 on Debian.
Byron commentedon Aug 29, 2013
Wow, exactly one year ago I said I had no time, and sadly this still didn't change. However, next year I could possibly get to it, there are many things I want to improve and do with git-python.
virgiliu commentedon Aug 30, 2013
That sounds great.
I've managed to make a workaround for the function that caused the "Too many open files" error using git commands.
Here's a simplified version of the code that caused the error:
After about 500 iterations it crashes with
Is there any way of getting the lastest commit from a directory or a file on a certain branch or tag besides using iter_commits ? I'd rather use objects instead of parsing raw git output and making my own dictionary with the values.
bochecha commentedon Aug 31, 2013
@virgiliu My personal workaround is to run the GitPython stuff (at least the ones which lead to this bug) in different processes. :-/
virgiliu commentedon Sep 2, 2013
@bochecha could you show me an example of that?
bochecha commentedon Sep 2, 2013
@virgiliu: sure. I just hope @Byron doesn't mind that we're using his bug tracker to discuss like this. :)
So, in one of my projects I'm using the following solution. I haven't pushed it yet (mostly because I'm not entirely satisfied with it) so I can't just give you a link to the code. You can assume that the code below is MIT-licensed.
So first, I have the following decorator:
Then, you just need to decorate a method with it, and it gets run as a subprocess!
In my case, I have a class
Foobarwhich does stuff on various Git repositories:Unfortunately the
do_stuff_on_repomethod goes through the wrong code path and I call it repeatedly (even on different repositories), so it leaks memory and fds.As a result, I'm just defining it in this way:
All I changed is that I decorated it, and now each call to the function will be run in its own process, so the leaking memory and fds get cleared when the process exits.
The only tricky part which is not handled above is what to do in case of an error: if
do_some_stuff_on_reporaises an exception for example, then subprocess just exits, without affecting the main process where your application is running.How to handle that properly is left as an exercise to the reader. (because that's the part I'm not very happy with :P)
4 remaining items
AnthonyBrunasso commentedon Aug 19, 2014
Ahhh, I am having the same issue. Would be nice to see a fix. :)
smn commentedon Oct 13, 2014
Is anyone working on this bug?
Byron commentedon Jan 7, 2015
The issue presented by @virgiliu seems to be fixed now (with the latest version of smmap 0.9.0), but by artificially constraining the file limit to 64, I can still see that pipes are leaked.

For some reason, repository instances are never freed, and I will have to figure out why that would happen.
Byron commentedon Jan 7, 2015
As
__del__()is not reliably called in python 3, the tests leak heavily as no repository it ever used gets cleaned up, leaving plenty of the processes shown above. Each of them comes with three pipes, quickly using up the processes resources.For example, it's possible to run all tests with
ulimit -n 64in python 2, but needsulimit -n 140in python 3.Byron commentedon Jan 7, 2015
It turned out I forgot to add a tear-down super class call, which was responsible for most of the test-leakage.
On the plus side, travis now checks for this as well.
An improved leak-test was added, which clearly shows that repositories dispose of their file handles correctly.
mricon commentedon Jan 7, 2015
Fantastic, thanks!
smithsp commentedon Oct 12, 2015
I am still seeing this problem in version 1.0.1. Here is a little test script:
At the same time, look at the number of open files with
lsof | grep ".git". I see that the number of open files increases by about 25. The only way to decrease the number of open files is to close the python session.(I should point out that I am running on Mac OSX, with a mostly MacPorts python installation.)
Byron commentedon Oct 16, 2015
I think it will work if you use the latest
HEADof this repository, which would bev1.0.2if I hadn't lost access to my pypi account.In
v1.0.1, you should be able to instantiate theRepotype withgit.Repo(path, odbt=GitCmdObjectDB)to alleviate the issue.If this works for you, I'd be glad if you could let me know here as well.
earwig commentedon Oct 29, 2015
I have some reports of the same issue in earwig/git-repo-updater#9, I think. Will ask people if GitPython HEAD fixes it.
trying to fix memory leak issue with instantiating repositories with …
fixing file descriptor leaks