Впервые я столкнулся с Memory Error, когда работал с огромным массивом ключевых слов. Там было около 40 млн. строк, воодушевленный своим гениальным скриптом я нажал Shift + F10 и спустя 20 секунд получил Memory Error.
Memory Error — исключение вызываемое в случае переполнения выделенной ОС памяти, при условии, что ситуация может быть исправлена путем удаления объектов. Оставим ссылку на доку, кому интересно подробнее разобраться с этим исключением и с формулировкой. Ссылка на документацию по Memory Error.
Если вам интересно как вызывать это исключение, то попробуйте исполнить приведенный ниже код.
print('a' * 1000000000000)
Почему возникает MemoryError?
В целом существует всего лишь несколько основных причин, среди которых:
- 32-битная версия Python, так как для 32-битных приложений Windows выделяет лишь 4 гб, то серьезные операции приводят к MemoryError
- Неоптимизированный код
- Чрезмерно большие датасеты и иные инпут файлы
- Ошибки в установке пакетов
Как исправить MemoryError?
Ошибка связана с 32-битной версией
Тут все просто, следуйте данному гайдлайну и уже через 10 минут вы запустите свой код.
Как посмотреть версию Python?
Идем в cmd (Кнопка Windows + R -> cmd) и пишем python. В итоге получим что-то похожее на
Python 3.8.8 (tags/v3.8.8:024d805, Feb 19 2021, 13:18:16) [MSC v.1928 64 bit (AMD64)]
Нас интересует эта часть [MSC v.1928 64 bit (AMD64)], так как вы ловите MemoryError, то скорее всего у вас будет 32 bit.
Как установить 64-битную версию Python?
Идем на официальный сайт Python и качаем установщик 64-битной версии. Ссылка на сайт с официальными релизами. В скобках нужной нам версии видим 64-bit. Удалять или не удалять 32-битную версию — это ваш выбор, я обычно удаляю, чтобы не путаться в IDE. Все что останется сделать, просто поменять интерпретатор.
Идем в PyCharm в File -> Settings -> Project -> Python Interpreter -> Шестеренка -> Add -> New environment -> Base Interpreter и выбираем python.exe из только что установленной директории. У меня это
C:/Users/Core/AppData/LocalPrograms/Python/Python38
Все, запускаем скрипт и видим, что все выполняется как следует.
Оптимизация кода
Пару раз я встречался с ситуацией когда мои костыли приводили к MemoryError. К этому приводили избыточные условия, циклы и буферные переменные, которые не удаляются после потери необходимости в них. Если вы понимаете, что проблема может быть в этом, вероятно стоит закостылить пару del, мануально удаляя ссылки на объекты. Но помните о том, что проблема в архитектуре вашего проекта, и по настоящему решить эту проблему можно лишь правильно проработав структуру проекта.
Явно освобождаем память с помощью сборщика мусора
В целом в 90% случаев проблема решается переустановкой питона, однако, я просто обязан рассказать вам про библиотеку gc. В целом почитать про Garbage Collector стоит отдельно на авторитетных ресурсах в статьях профессиональных программистов. Вы просто обязаны знать, что происходит под капотом управления памятью. GC — это не только про Python, управление памятью в Java и других языках базируется на технологии сборки мусора. Ну а вот так мы можем мануально освободить память в Python:
Last Updated on March 13, 2023 by Prepbytes
In Python, errors are exceptions that occur when the interpreter encounters a problem while running a program. These exceptions can be caught and managed using try-except blocks, allowing a program to handle problems gracefully and recover from them. There are many different types of errors in Python, each with its own meaning and origin. Python memory error is one such error. Let’s take a closer look into the python memory error.
What is Python Memory Error?
A Python memory error occurs when a Python program runs out of memory while attempting to allocate space for new objects. This error usually occurs when a program attempts to generate a massive data structure, such as a list or dictionary, or when it creates little objects continuously until the available memory is consumed.
When a memory error occurs, Python throws a MemoryError exception, indicating that the needed memory could not be allocated. This exception can be handled gracefully by implementing a try-except block, allowing the program to handle any necessary cleanup or error reporting.
Here’s an example of a MemoryError in Python:
# Creating a large list
my_list = [i for i in range(1000000000)]This code creates a list of one billion integers. Depending on the available memory on your system, this can quickly consume all available memory and raise a MemoryError. The error message might look something like this:
Traceback (most recent call last):
File "", line 1, in
MemoryErrorTo avoid this error, you can use a generator expression instead of a list comprehension, as generators create values on-the-fly and don’t store them in memory all at once. Here’s an example:
# Using a generator expression
my_generator = (i for i in range(1000000000))This code creates a generator expression that generates the integers from 0 to 999,999,999 on-the-fly, rather than creating a list that stores all the integers in memory. This can help avoid the MemoryError.
Types of Python Memory Error
Here are some more typical ways that Python memory errors might be classified:
1. Out of Memory Error
An Out of Memory Error in Python occurs when a program tries to allocate more memory than is available on the system. This can happen when a user is working with large datasets, images, or other memory-consumption tasks. For example, suppose a user tries to create a list of one billion integers on a system with limited memory, it may run out of memory and throws an Out of Memory Error.
2. Stack Overflow Error
A Stack Overflow Error occurs when a program’s call stack becomes too deep, causing a stack overflow. This can happen when using a recursive function. For example, if a program uses a recursive function with no base case, it may run out of stack space and raise a Stack Overflow Error.
3. Memory Leak Error
A Memory Leak Error occurs when a program continuously allocates memory without releasing it, leading to a gradual increase in memory usage over time. This can happen when a program repeatedly allocates memory to new objects or data structures without properly releasing old memory, or when there are circular references between objects that prevent them from being garbage-collected.
Ways to Handle Python Memory Error
These are some ways to handle Python memory errors:
1. Appropriate Python Set-up
This simplest, but perhaps least intuitive, answer to a MemoryError is actually related to a potential problem with your Python setup. If you install the 32-bit version of Python on a 64-bit system, you will have highly limited access to the system’s RAM. This restricted access may result in MemoryErrors on programs that your computer would normally be able to handle.
2. Use Generators or Iterators
Instead of putting all of the data into memory at once, use generators or iterators to process data in smaller chunks. This can help to reduce memory use and avoid memory faults. For example, in a function, use yield to generate data one item at a time.
def generate_data():
for i in range(1000000):
yield i
for data in generate_data():
process_data(data)3. Use a Database or External Storage
If the data set is too large to fit in memory, consider storing it in a database or an file. This allows the program to access the data without needing to load it all into memory all at once. Here’s an example using the pandas library to read a large CSV file:
# Importing the pandas library
import pandas as pd
# Reading a large CSV file
df = pd.read_csv('large_file.csv')4. Improve Memory Consumption
Review the code and look for places where memory usage can be improved. For example, instead of copying lists, use slicing, map, or filter built-in functions instead of loops.
5. Catch and Handle Exceptions
Catch MemoryError exceptions and handle them in a way that allows the program to continue functioning. For example, log the error and try again later, or display an error message to the user and politely exit the program. Here’s an code snippet to demonstrate this:
try:
# Code that might raise a MemoryError
except MemoryError:
# Handle the error6. Memory Profiler to Identify Memory Leaks
A memory profiler is a tool that can be used to identify memory leaks in Python programs. Memory leaks occur when a program does not release memory that is no longer needed, causing memory usage to increase over time. Memory profilers help identify the parts of a program that are responsible for memory leaks, allowing developers to address these issues and improve the program’s memory efficiency.
Conclusion
Python memory error occurs when a program tries to use more memory than is available. This can happen due to various reasons, such as not deallocating the memory of unused objects or improper memory usage. To avoid and fix memory errors, it’s important to be aware of the memory usage of our programs and take necessary steps to manage our memory, such as using generators in place of lists, deleting unused objects, and using external storage. By being mindful of memory usage, we can write more efficient Python programs.
FAQs Related to Python Memory Error
Here are some frequently asked questions (FAQs) about Python memory errors:
Q1 – What is a memory error in python?
Ans – A Python memory error occurs when a program tries to allocate more memory than the system can provide. This can happen when a program tries to store too much data in memory or when a program creates too many objects.
Q2 – What causes memory errors in python?
Ans – Python memory errors can be caused by a variety of factors, including a lack of available memory on the system, inefficient code that creates too many objects, or a data set that is too large to fit into memory.
Q3 – How to handle Python memory errors?
Ans – We can handle Python memory errors by catching the MemoryError exception and taking appropriate action, such as freeing up memory or terminating the program. Additionally, we can also use profiling tools to identify memory usage in our code.
Q4 – What should I do if I encounter a Python memory error?
Ans – If you encounter a Python memory error, try to identify the source of the error by using profiling tools or by analyzing your python code. Once you have identified the source, take appropriate action to reduce memory usage, such as using memory-efficient data structures or deleting objects that are no longer needed. If you are working with large data sets, consider using external storage to store them.
Updated April 3, 2023
Introduction to Python Memory Error
Memory Error is a kind of error in python that occurs when where the memory of the RAM we are using could not support the execution of our code since the memory of the RAM is smaller and the code we are executing requires more than the memory of our existing RAM, this often occurs when a large volume of data is fed to the memory and when the program has run out of memory in processing the data.
Syntax of Python Memory Error
When performing an operation that generates or using a big volume of data, it will lead to a Memory Error.
Code:
## Numpy operation which return random unique values
import numpy as np
np.random.uniform(low=1,high=10,size=(10000,100000))Output:
For the same function, let us see the Name Error.
Code:
## Functions which return values
def calc_sum(x,y):
op = x + y
return(op)The numpy operation of generating random numbers with a range from a low value of 1 and highest of 10 and a size of 10000 to 100000 will throw us a Memory Error since the RAM could not support the generation of that large volume of data.
How does Memory Error Works?
Most often, Memory Error occurs when the program creates any number of objects, and the memory of the RAM runs out. When working on Machine Learning algorithms most of the large datasets seems to create Memory Error. Different types of Memory Error occur during python programming. Sometimes even if your RAM size is large enough to handle the datasets, you will get a Memory Error. This is due to the Python version you might be using some times; 32-bit will not work if your system is adopted to a 64-bit version. In such cases, you can go uninstall 32-bit python from your system and install the 64-bit from the Anaconda website. When you are installing different python packages using the pip command or other commands may lead to improper installation and throws a Memory Error.
In such cases, we can use the conda install command in python prompt and install those packages to fix the Memory Error.
Example:
Another type of Memory Error occurs when the memory manager has used the Hard disk space of our system to store the data that exceeds the RAM capacity. Upon working, the computer stores all the data and uses up the memory throws a Memory Error.
Avoiding Memory Errors in Python
The most important case for Memory Error in python is one that occurs during the use of large datasets. Upon working on Machine Learning problems, we often come across large datasets which, upon executing an ML algorithm for classification or clustering, the computer memory will instantly run out of memory. We can overcome such problems by executing Generator functions. It can be used as a user-defined function that can be used when working with big datasets.
Generators allow us to efficiently use the large datasets into many segments without loading the complete dataset. Generators are very useful in working on big projects where we have to work with a large volume of data. Generators are functions that are used to return an iterator. Iterators can be used to loop the data over. Writing a normal iterator function in python loops the entire dataset and iters over it. This is where the generator comes in handy it does not allow the complete dataset to loop over since it causes a Memory Error and terminates the program.
The generator function has a special characteristic from other functions where a statement called yield is used in place of the traditional return statement that returns the output of the function.
A sample Generator function is given as an example:
Code:
def sample_generator():
for i in range(10000000):
yield i
gen_integ= sample_generator()
for i in gen_integ:
print(i)Output:
In this sample generator function, we have generated integers using the function sample generator, which is assigned to the variable gen_integ, and then the variable is iterated. This allows us to iter over one single value at a time instead of passing the entire set of integers.
In the sample code given below, we have tried to read a large dataset into small bits using the generator function. This kind of reading would allow us to process large data in a limited size without using up the system memory completely.
Code:
def readbits(filename, mode="r", chunk_size=20):
with open(filename, mode) as f:
while True:
data = f.read(chunk_size)
if not data:
break
yield data
def main():
filename = "C://Users//Balaji//Desktop//Test"
for bits in readbits(filename):
print(bits) Output:
There is another useful technique that can be used to free memory while we are working on a large number of objects. A simple way to erase the objects that are not referenced is by using a garbage collector or gc statement.
Code:
import gc
gc.collect()The import garbage collector and gc.collect() statement allows us to free the memory by removing the objects which the user does not reference.
There are additional ways in which we can manage the memory of our system CPU where we can write code to limit the CPU usage of memory.
Code:
import resource
def limit_memory(Datasize):
min_, max_ = resource.getrlimit(resource.RLIMIT_AS)
resource.setrlimit(resource.RLIMIT_AS, (Datasize, max_)) This allows us to manage CPU usage to prevent Memory Error.
Some of the other techniques that can be used to overcome the Memory Error are to limit our sample size of we are working on, especially while performing complex machine learning algorithms. Or we could update our system with more memory, or we can use the cloud services like Azure, AWS, etc. that provides the user with strong computing capabilities.
Another way is to use the Relational Database Management technique where open-source databases like MySQL are available free of cost. It can be used to store large volumes of data; also, we can adapt to big data storage services to effectively work with large volumes.
Conclusion
In detail, we have seen the Memory Error that occurs in the Python programming language and the techniques to overcome the Name Error. The main take away to remember in python Memory Error is the memory usage of our RAM where the operations are taking place, and efficiently using the above-mentioned techniques will allow us to overcome the Memory Error.
Recommended Articles
This is a guide to Python Memory Error. Here we discuss the introduction, working and avoiding memory errors in python, respectively. You may also have a look at the following articles to learn more –
- Python IOError
- Custom Exception in Python
- Python AssertionError
- Python Object to String
What is Memory Error?
Python Memory Error or in layman language is exactly what it means, you have run out of memory in your RAM for your code to execute.
When this error occurs it is likely because you have loaded the entire data into memory. For large datasets, you will want to use batch processing. Instead of loading your entire dataset into memory you should keep your data in your hard drive and access it in batches.
A memory error means that your program has run out of memory. This means that your program somehow creates too many objects. In your example, you have to look for parts of your algorithm that could be consuming a lot of memory.
If an operation runs out of memory it is known as memory error.
Types of Python Memory Error
Unexpected Memory Error in Python
If you get an unexpected Python Memory Error and you think you should have plenty of rams available, it might be because you are using a 32-bit python installation.
The easy solution for Unexpected Python Memory Error
Your program is running out of virtual address space. Most probably because you’re using a 32-bit version of Python. As Windows (and most other OSes as well) limits 32-bit applications to 2 GB of user-mode address space.
We Python Pooler’s recommend you to install a 64-bit version of Python (if you can, I’d recommend upgrading to Python 3 for other reasons); it will use more memory, but then, it will have access to a lot more memory space (and more physical RAM as well).
The issue is that 32-bit python only has access to ~4GB of RAM. This can shrink even further if your operating system is 32-bit, because of the operating system overhead.
For example, in Python 2 zip function takes in multiple iterables and returns a single iterator of tuples. Anyhow, we need each item from the iterator once for looping. So we don’t need to store all items in memory throughout looping. So it’d be better to use izip which retrieves each item only on next iterations. Python 3’s zip functions as izip by default.
Must Read: Python Print Without Newline
Python Memory Error Due to Dataset
Like the point, about 32 bit and 64-bit versions have already been covered, another possibility could be dataset size, if you’re working with a large dataset. Loading a large dataset directly into memory and performing computations on it and saving intermediate results of those computations can quickly fill up your memory. Generator functions come in very handy if this is your problem. Many popular python libraries like Keras and TensorFlow have specific functions and classes for generators.
Python Memory Error Due to Improper Installation of Python
Improper installation of Python packages may also lead to Memory Error. As a matter of fact, before solving the problem, We had installed on windows manually python 2.7 and the packages that I needed, after messing almost two days trying to figure out what was the problem, We reinstalled everything with Conda and the problem was solved.
We guess Conda is installing better memory management packages and that was the main reason. So you can try installing Python Packages using Conda, it may solve the Memory Error issue.
Most platforms return an “Out of Memory error” if an attempt to allocate a block of memory fails, but the root cause of that problem very rarely has anything to do with truly being “out of memory.” That’s because, on almost every modern operating system, the memory manager will happily use your available hard disk space as place to store pages of memory that don’t fit in RAM; your computer can usually allocate memory until the disk fills up and it may lead to Python Out of Memory Error(or a swap limit is hit; in Windows, see System Properties > Performance Options > Advanced > Virtual memory).
Making matters much worse, every active allocation in the program’s address space can cause “fragmentation” that can prevent future allocations by splitting available memory into chunks that are individually too small to satisfy a new allocation with one contiguous block.
1 If a 32bit application has the LARGEADDRESSAWARE flag set, it has access to s full 4gb of address space when running on a 64bit version of Windows.
2 So far, four readers have written to explain that the gcAllowVeryLargeObjects flag removes this .NET limitation. It does not. This flag allows objects which occupy more than 2gb of memory, but it does not permit a single-dimensional array to contain more than 2^31 entries.
How can I explicitly free memory in Python?
If you wrote a Python program that acts on a large input file to create a few million objects representing and it’s taking tons of memory and you need the best way to tell Python that you no longer need some of the data, and it can be freed?
The Simple answer to this problem is:
Force the garbage collector for releasing an unreferenced memory with gc.collect().
Like shown below:
import gc
gc.collect()
Memory error in Python when 50+GB is free and using 64bit python?
On some operating systems, there are limits to how much RAM a single CPU can handle. So even if there is enough RAM free, your single thread (=running on one core) cannot take more. But I don’t know if this is valid for your Windows version, though.
How do you set the memory usage for python programs?
Python uses garbage collection and built-in memory management to ensure the program only uses as much RAM as required. So unless you expressly write your program in such a way to bloat the memory usage, e.g. making a database in RAM, Python only uses what it needs.
Which begs the question, why would you want to use more RAM? The idea for most programmers is to minimize resource usage.
if you wanna limit the python vm memory usage, you can try this:
1、Linux， ulimit command to limit the memory usage on python
2、you can use resource module to limit the program memory usage;
if u wanna speed up ur program though giving more memory to ur application, you could try this:
1\threading, multiprocessing
2\pypy
3\pysco on only python 2.5
How to put limits on Memory and CPU Usage
To put limits on the memory or CPU use of a program running. So that we will not face any memory error. Well to do so, Resource module can be used and thus both the task can be performed very well as shown in the code given below:
Code #1: Restrict CPU time
# importing libraries
import signal
import resource
import os
# checking time limit exceed
def time_exceeded(signo, frame):
print("Time's up !")
raise SystemExit(1)
def set_max_runtime(seconds):
# setting up the resource limit
soft, hard = resource.getrlimit(resource.RLIMIT_CPU)
resource.setrlimit(resource.RLIMIT_CPU, (seconds, hard))
signal.signal(signal.SIGXCPU, time_exceeded)
# max run time of 15 millisecond
if __name__ == '__main__':
set_max_runtime(15)
while True:
pass
Code #2: In order to restrict memory use, the code puts a limit on the total address space
# using resource import resource def limit_memory(maxsize): soft, hard = resource.getrlimit(resource.RLIMIT_AS) resource.setrlimit(resource.RLIMIT_AS, (maxsize, hard))
Ways to Handle Python Memory Error and Large Data Files
1. Allocate More Memory
Some Python tools or libraries may be limited by a default memory configuration.
Check if you can re-configure your tool or library to allocate more memory.
That is, a platform designed for handling very large datasets, that allows you to use data transforms and machine learning algorithms on top of it.
A good example is Weka, where you can increase the memory as a parameter when starting the application.
2. Work with a Smaller Sample
Are you sure you need to work with all of the data?
Take a random sample of your data, such as the first 1,000 or 100,000 rows. Use this smaller sample to work through your problem before fitting a final model on all of your data (using progressive data loading techniques).
I think this is a good practice in general for machine learning to give you quick spot-checks of algorithms and turnaround of results.
You may also consider performing a sensitivity analysis of the amount of data used to fit one algorithm compared to the model skill. Perhaps there is a natural point of diminishing returns that you can use as a heuristic size of your smaller sample.
3. Use a Computer with More Memory
Do you have to work on your computer?
Perhaps you can get access to a much larger computer with an order of magnitude more memory.
For example, a good option is to rent compute time on a cloud service like Amazon Web Services that offers machines with tens of gigabytes of RAM for less than a US dollar per hour.
4. Use a Relational Database
Relational databases provide a standard way of storing and accessing very large datasets.
Internally, the data is stored on disk can be progressively loaded in batches and can be queried using a standard query language (SQL).
Free open-source database tools like MySQL or Postgres can be used and most (all?) programming languages and many machine learning tools can connect directly to relational databases. You can also use a lightweight approach, such as SQLite.
5. Use a Big Data Platform
In some cases, you may need to resort to a big data platform.
Summary
In this post, you discovered a number of tactics and ways that you can use when dealing with Python Memory Error.
Are there other methods that you know about or have tried?
Share them in the comments below.
Have you tried any of these methods?
Let me know in the comments.
If your problem is still not solved and you need help regarding Python Memory Error. Comment Down below, We will try to solve your issue asap.
A MemoryError means that the interpreter has run out of memory to allocate to your Python program. This may be due to an issue in the setup of the Python environment or it may be a concern with the code itself loading too much data at the same time.
An Example of MemoryError
To have a look at this error in action, let’s start with a particularly greedy piece of code. In the code below, we start with an empty array and use nested arrays to add strings to it. In this case, we use three levels of nested arrays, each with a thousand iterations. This means at the end of the program, the array s has 1,000,000,000 copies of the string «More.«
s = []
for i in range(1000):
for j in range(1000):
for k in range(1000):
s.append("More")Output
As you might expect, these million strings are a bit much for, let’s say, a laptop to handle. The following error is printed out:
C:\code\Python\MemErr\venv\3K\Scripts\python.exe C:/code/python/MemErr/main.py
Traceback (most recent call last):
File "C:/code/python/MemErr/main.py", line 6, in <module>
s.append("More")
MemoryErrorIn this case, the traceback is relatively simple as there are no libraries involved in this short program. After the traceback showing the exact function call which caused the issue, we see the simple but direct MemoryError.
Two Ways to Handle A MemoryError in Python
Appropriate Python Set-up
This simplest but possibly least intuitive solution to a MemoryError actually has to do with a potential issue with your Python setup. In the event that you have installed the 32-bit version of Python on a 64-bit system, you will have extremely limited access to the system’s memory. This restricted access may cause MemoryErrors on programs that your computer would normally be able to handle.
Attention to Large Nested Loops
If your installation of Python is correct and these issues still persist, it may be time to revisit your code. Unfortunately, there is no cut and dry way to entirely remove this error outside of evaluating and optimizing your code. Like in the example above, pay special attention to any large or nested loops, along with any time you are loading large datasets into your program in one fell swoop.
In these cases, the best practice is often to break the work into batches, allowing the memory to be freed in between calls. As an example, in the code below, we have broken out earlier nested loops into 3 separate loops, each running for 333,333,333 iterations. This program still goes through one million iterations but, as the memory can be cleared through the process using a garbage collection library, it no longer causes a MemoryError.
An Example of Batching Nested Loops
import gc
s = []
t = []
u = []
for i in range(333333333):
s.append("More")
gc.collect()
for j in range(333333333):
t.append("More")
gc.collect()
for k in range(333333334):
u.append("More")
gc.collect()How to Avoid a MemoryError in Python
Python’s garbage collection makes it so that you should never encounter issues in which your RAM is full. As such, MemoryErrors are often indicators of a deeper issue with your code base. If this is happening, it may be an indication that more code optimization or batch processing techniques are required. Thankfully, these steps will often show immediate results and, in addition to avoiding this error, will also vastly shorten the programs’ runtime and resource requirements.
Track, Analyze and Manage Errors With Rollbar
Managing errors and exceptions in your code is challenging. It can make deploying production code an unnerving experience. Being able to track, analyze, and manage errors in real-time can help you proceed with more confidence. Rollbar automates error monitoring and triaging, making fixing Python errors easier than ever. Sign Up Today!