Ошибка субд cannot allocate memory for output buffer

What type of bug is this?

Unexpected error

What subsystems and features are affected?

Distributed hypertable, Multi-node

What happened?

Encountered an error:

ERROR: [dn1]: cannot allocate memory for output buffer

after upgrading Timescale DB from 2.7.2 to 2.8.1.
This error occurs when writing to a distributed hyper table. If the size of one transaction in the total_bytes chunk < 2 GB, then the transaction is completed successfully. If exceeded, it is interrupted by an error:

ERROR: [dn1]: cannot allocate memory for output buffer

After rolling back TimescaleDB to version 2.7.2, this error is not observed.

TimescaleDB version affected

2.8.1

PostgreSQL version used

14.6

What operating system did you use?

Ubuntu 20.04 x64

What installation method did you use?

Deb/Apt

What platform did you run on?

Other

Relevant log output and stack trace

2022-11-28 14:36:55.631 UTC [1366403] user@test_db WARNING:  only one data node was assigned to the hypertable
2022-11-28 14:36:55.631 UTC [1366403] user@test_db DETAIL:  A distributed hypertable should have at least two data nodes for best performance.
2022-11-28 14:36:55.631 UTC [1366403] user@test_db HINT:  Grant USAGE on data nodes and attach them to the hypertable.
2022-11-28 14:41:48.985 UTC [1366441] user@test_db ERROR:  [dn1]: cannot allocate memory for output buffer
	
2022-11-28 14:41:48.985 UTC [1366441] user@test_db STATEMENT:  INSERT INTO test_distributed_hypertable 
	SELECT '2050-01-01'::date,series,0,0,0,0,0,0,0,0,0,0,0,0,0
	FROM generate_series(1, 20000000, 1) series
2022-11-28 14:41:49.213 UTC [1366327] user@test_db LOG:  could not receive data from client: Connection reset by peer
2022-11-28 14:41:49.213 UTC [1366441] user@test_db LOG:  could not receive data from client: Connection reset by peer
2022-11-28 14:41:49.215 UTC [1366328] user@test_db LOG:  could not receive data from client: Connection reset by peer
2022-11-28 14:46:48.579 UTC [1366871] user@test_db ERROR:  [dn1]: cannot allocate memory for output buffer
	
2022-11-28 14:46:48.579 UTC [1366871] user@test_db STATEMENT:  INSERT INTO test_distributed_hypertable 
	SELECT '2050-01-01'::date,series,0,0,0,0,0,0,0,0,0,0,0,0,0
	FROM generate_series(1, 20000000, 1) series
2022-11-28 14:46:48.839 UTC [1366868] user@test_db LOG:  could not receive data from client: Connection reset by peer
2022-11-28 14:46:48.839 UTC [1366869] user@test_db LOG:  could not receive data from client: Connection reset by peer
2022-11-28 14:46:48.839 UTC [1366871] user@test_db LOG:  could not receive data from client: Connection reset by peer

How can we reproduce the bug?

CREATE TABLE test_distributed_hypertable (
	create_date date NOT NULL,
	product_id int8 NOT NULL,
	size_id int8 NOT NULL,
	warehouse_id int8 NOT NULL,
	quantity int4 NOT NULL,
	diff_quantity int4 NOT NULL,
	orders int4 NOT NULL,
	incomes  int4 NOT NULL,
	"returns" int4 NOT NULL,
	proceeds float8 NOT NULL,
	price float8 NOT NULL,
	old_price float8 NOT NULL,
	discount int4 NOT NULL DEFAULT 0,
	basic_discount int4 NOT NULL DEFAULT 0,
	promo_discount int4 NOT NULL DEFAULT 0,
	CONSTRAINT test_distributed_hypertable_pk PRIMARY KEY (product_id, size_id, warehouse_id, create_date)
);
SELECT create_distributed_hypertable('test_distributed_hypertable', 'create_date', chunk_time_interval => '1d'::interval, data_nodes => '{ "dn1"}');


INSERT INTO test_distributed_hypertable 
SELECT '2050-01-01'::date,series,0,0,0,0,0,0,0,0,0,0,0,0,0
FROM generate_series(1, 20000000, 1) series;

Pete Erickson <redlamb(at)redlamb(dot)net> writes:
> I am looking for some help regarding an python OperationalError that I
> recently received while executing a python script using sqlalchemy and
> psycopg2. The python script parses an xml file stored on a networked
> drive and enters the information into a pgsql database. Sometimes
> these xml files reference a binary file which is also located on the
> networked drive. These files are subsequently read in and stored in a
> table along with the file’s md5. The binary data is stored within a
> bytea column. This script worked pretty well until recently when it
> came across a binary file about 258MB in size. While reading the file
> off the networked drive I received an OperationalError indicating that
> it was unable to allocate memory for the output buffer. My initial
> guess was that it ran out of memory, but according to the task manager
> the machine had close to 2GB free when the error occurred.

Out of memory is probably exactly right. The textual representation of
arbitrary bytea data is normally several times the size of the raw bits
(worst case is 5x bigger, typical case perhaps half that). In addition
to that you have to consider that there are likely to be several copies
of the string floating around in your process’ memory space. If you’re
doing this in a 32bit environment it doesn’t surprise me at all that
258MB of raw data would exhaust available memory.

Going to a 64bit implementation would help some, but I’m not sure that
that’s an available option for you on Windows, and anyway it doesn’t
eliminate the problem completely. If you want to process really large
binary files you’re going to need to divide them into segments.

regards, tom lane

1C
 
Загрузка базы на PostGre
0

Stepa86

09.11.07

13:05

Есть серверный вариант УПП на PostGreSQL, выгрузил его в dt, в файловом варианте загрузил, объеденил с доработанной базой УПП (.cf загрузил). Создал пустую базу на сервере, стал туда загружать dt. вылетело «Ошибка СУБД: cannot allocate memory for output buffer». Конфигуратор вылетает, при повторном запуске пишет — конфигурационная база разрушена. Как можно обойти??? (объединять сразу в серверной mfpt тоже не получается — другая ошибка, но тоже потом все разрушено)

1

Stepa86

09.11.07

14:38

Неужели с такой проблемой столкнулся только я и какой то чел с Украины? И то его решение было перейти на другую СУБД

Требовать и эффективности, и гибкости от одной и той же программы — все равно, что искать очаровательную и скромную жену… по-видимому, нам следует остановиться на чем-то одном из двух. Фредерик Брукс-младший

 Like 

#0
by Doomer

Файл не обнаружен ‘e0666db2-45d6-49b4-a200-061c6ba7d569.64231292-7296-4c35-9575-2cf8315405c9.new’ по причине: Ошибка СУБД: cannot allocate memory for output buffer Что за ошибка? Как с ней справиться?

 Like 

#1
by mikecool

памяти не хватает

 Like 

#2
by mishaPH

учимся правильно создавать ветки и не вандалим.

 Like 

#3
by Doomer

Это проделки postgres. Ошибка выходит при обновлении бП.

Тэги: Админ

Ответить:

Комментарии доступны только авторизированным пользователям

Регистрация

Похожие вопросы 1С

  • Ошибка СУБД, внутрення ошибка
  • Странная ошибка «ошибка формата потока» с вылетом из сеанса
  • Ошибка выполнения запроса «Ошибка при выполнении операции над данными: Ошибка SQL: Де
  • v7: Ошибка СУБД.Внутренняя ошибка dbeng 8
  • При отправке сообщения произошла ошибка : Общая ошибка MAPI
  • Помогите восстановить базу после сбоя. Ошибка СУБД: Ошибка SQL….
  • v8: Ошибка СУБД. Внутренняя ошибка компоненты DBENG8 при обмене данными
  • Ошибка при попытке выгрузить данные из УТ в БП. интересная необычная ошибка
  • Ошибка при выгрузке базы: «Ошибка СУБД: out of memory for query result»
  • Ошибка при вызове web-сервиса 1C 8.3 (Внутренняя ошибка. Ошибка инициализации б)
  • Ошибка при вызове метода контекста (Записать) Ошибка при выполнении файловой опе
  • Ошибка СУБД: Microsoft ole db provider for sql server

В этой группе 1С

  • Где прочитать про работу в 1с77 с xml
  • 7.7: Календари в ЗиК
  • как спозиционироваться на записи регистра и установить снять в нем значения?
  • Зик 7.7 больничный лист расчет МРОТ
  • v8: УПП — не могу найти отчет по МПЗ в разрезе ГТД?
  • Получить движения документа в подписке
  • Загрузка из Excel в табличную часть
  • ОкончаниеВводаТекста и АвтоПодбор
  • v8: УПП — Док. Принятие к учету ОС — не заполняется регистр Учет затрат?
  • deb пакеты, установка
  • Код справочника стал не уникальным. Договоры контрагентов. Задвоения кодов нет.
  • ЗУП 2.5 Кадровое планирование
  • Как в ЗУП ввести переходящий больничный ?
  • Загрузка номенклатуры с чужого сайта в 1С Управление торговлей 8.2
  • Фотоаппарат вместо сканера? реально или нет?
  • УПП: ошибка в обработке Подготовка данных ПФР
  • Не подключается к серверу терминалов.
  • Доступ к метаданным в управляемой форме
  • ЗУП 2.5.32.5, Внешняя форма ФСС для 2.5.33.3
  • V77Exp.ert, V77IMP.ert

Avatar

Hi

So I was developing a module that exports every products barcode and image to an Excel, and that then returns the an Excel file. This works fine when there’s just a couple pictures, but when I tried running the method on a bigger db with tons more pictures I got an error.

Here is the code that’s being run:


class ExportProductImage(models.TransientModel):
_name = «export.product.image»
_description = «Export product image»
datas = fields.Binary(‘File’, readonly=True)
datas_fname = fields.Char(‘Filename’, readonly=True)
def generate_excel_product_image(self):
filename = ‘product_images’
fp = BytesIO()
book = xlsxwriter.Workbook(fp)
sheet1 = book.add_worksheet(«Afbeeldingen»)
sheet1.set_landscape()
sheet1.set_default_row(230)
sheet1.set_column(‘B:B’, 73)
sheet1.set_row(0, 20)
sheet1.write(‘A1’, ‘Barcode’)
sheet1.write(‘B1’, ‘Afbeelding’)
product_templates = self.env[‘product.template’].search([]) # image_1920
row = 2
for temp in product_templates:
if temp.image_1024:
sheet1.write(‘A’+str(row), «‘» + str(temp.barcode))
image = BytesIO(base64.b64decode(temp.image_1024))
sheet1.insert_image(‘B’ + str(row), «img» + str(row), {‘x_scale’: 0.5, ‘y_scale’: 0.5, ‘image_data’: image, ‘object_position’: 1})
row += 1
book.close()
out = base64.encodestring(fp.getvalue())
self.write({‘datas’: out, ‘datas_fname’: filename})
fp.close()
filename += ‘%2Exlsx’
return {
‘type’: ‘ir.actions.act_url’,
‘target’: ‘new’,
‘url’: ‘web/content/?model=’ + self._name + ‘&id=’ + str(
self.id) + ‘&field=datas&download=true&filename=’ + filename,
}

Here is the traceback:

Fout:
Odoo Server Error

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/odoo/sql_db.py", line 225, in execute
    res = self._obj.execute(query, params)
psycopg2.OperationalError: cannot allocate memory for output buffer


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/odoo/loglevels.py", line 92, in ustr
    return text_type(value)
MemoryError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/odoo/http.py", line 656, in _handle_exception
    return super(JsonRequest, self)._handle_exception(exception)
  File "/usr/lib/python3/dist-packages/odoo/http.py", line 314, in _handle_exception
    raise pycompat.reraise(type(exception), exception, sys.exc_info()[2])
  File "/usr/lib/python3/dist-packages/odoo/tools/pycompat.py", line 87, in reraise
    raise value
  File "/usr/lib/python3/dist-packages/odoo/http.py", line 698, in dispatch
    result = self._call_function(**self.params)
  File "/usr/lib/python3/dist-packages/odoo/http.py", line 346, in _call_function
    return checked_call(self.db, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/odoo/service/model.py", line 97, in wrapper
    return f(dbname, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/odoo/http.py", line 339, in checked_call
    result = self.endpoint(*a, **kw)
  File "/usr/lib/python3/dist-packages/odoo/http.py", line 941, in __call__
    return self.method(*args, **kw)
  File "/usr/lib/python3/dist-packages/odoo/http.py", line 519, in response_wrap
    response = f(*args, **kw)
  File "/usr/lib/python3/dist-packages/odoo/addons/web/controllers/main.py", line 966, in call_button
    action = self._call_kw(model, method, args, {})
  File "/usr/lib/python3/dist-packages/odoo/addons/web/controllers/main.py", line 954, in _call_kw
    return call_kw(request.env[model], method, args, kwargs)
  File "/usr/lib/python3/dist-packages/odoo/api.py", line 759, in call_kw
    return _call_kw_multi(method, model, args, kwargs)
  File "/usr/lib/python3/dist-packages/odoo/api.py", line 746, in _call_kw_multi
    result = method(recs, *args, **kwargs)
  File "/mnt/repo/custom/report_stock_wizard/wizard/export_image.py", line 42, in generate_excel_product_image
    self.write({'datas': out, 'datas_fname': filename})
  File "/usr/lib/python3/dist-packages/odoo/models.py", line 3324, in write
    self._write(store_vals)
  File "/usr/lib/python3/dist-packages/odoo/models.py", line 3417, in _write
    cr.execute(query, params + [sub_ids])
  File "/usr/lib/python3/dist-packages/odoo/sql_db.py", line 148, in wrapper
    return f(self, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/odoo/sql_db.py", line 228, in execute
    _logger.error("bad query: %s\nERROR: %s", ustr(self._obj.query or query), e)
  File "/usr/lib/python3/dist-packages/odoo/loglevels.py", line 94, in ustr
    raise UnicodeError('unable to convert %r' % (value,))
MemoryError

Anyone know how I could solve this?

For those wondering, these are in my config file:

limit_memory_hard = 2684354560
limit_memory_soft = 2147483648
limit_request = 8192
limit_time_cpu = 120
limit_time_real = 9999
limit_time_real_cron = -1
longpolling_port = 8072
max_cron_threads = 9
workers = 0

When I put more then 0 workers, I get a bus error so that’s another problem.

Понравилась статья? Поделить с друзьями:
  • Ошибка субару с0029
  • Ошибка субд interface 0c733a7c 2a1c 11ce ade5 00aa0044773d
  • Ошибка субд database не пригоден для использования postgresql
  • Ошибка субд could not open file
  • Ошибка субару с0024