Generating very large XML files in Python? -

-

Does anyone know memory efficient ways to generate very large XML files (eg 100-500 MiB) in Python? is?

I am using lxml , but the memory usage is through the roof.

Maybe you can use the templing engine instead of creating / creating XML?

For example, XML-based and supports streaming output. A very basic example: genshi.template import markup template from 

  tpl_xml = '' '& lt; Doc xmlns: py = "http://genshi.edgewall.org/" & gt; & Lt; P py: for = "I am in data" & gt; $ {I} & lt; / P & gt; & Lt; / Doc & gt; F: stream.render (out = f) as' open '(' output.xml ',' w ') with' '' tpl = markupTemplate (tpl_xml) stream = tpl.generate (data = xrange (10000000)) < / Code>

It may take a little while, but memory usage is low.

Not for the templateing engine (for example, "natively" xml), but very fast:

From Mako.template import contact context by mako.runtime import tpl_xml = '' '& Lt; Doc & gt; % I for data: & lt; P & gt; $ {I} & lt; / P & gt; % Endfor & lt; / Doc & gt; '' Tpl = Template (tpl_xml) '(' output.xml ',' w ') as in c: f: ctx = context (f, data = xrange (10000000)) tpl.render_context (ctx)

The last example went to my laptop for about 20 seconds, created a (very simple) 151 MB XML file, there was no memory problem (according to the Windows Task Manager It remained constant at about 10 mm)

Depending on your needs, using this SX etc. is a friendly and quick to generate XML The method can be ... See docs to see if you can do with these engines (like others, I chose these two as examples)


Comments

Post a Comment

Popular posts from this blog

windows - Heroku throws SQLITE3 Read only exception -

-
After I deploy the app to Heroku, I have to run migration scripts and get this error message ... ITES \ padrino \ prophetmargin & gt; Heroku Rack AR: Rack mutated! SQLite3 :: ReadOnlyException: only one try to write database to read: CREATE TABLE "schema_migrations" ( "version" varchar (255) NOT NULL) /disk1/home/slugs/215264_925fd2c_65a3/mnt/.bundle/gems/gems/ padrino-core- is 0.9.11 / lib / padrino-core / cli / rake.rb: 9: in `init ' how can this be? I also tried to run Heroku dbpush SQLite: //db/my-db.db and that did not work Horoku sqlite3 but using postgres does not do. I'm not sure why you are receiving this error, however, as I have used sqlite3 in development and when pushing it into its travel, they do some magic which migrates to postgrads. I'm absolutely sure how Heroku does 'backwash' this database but it seems that this is happening to you because it's the SQLite db file which is obviously due to Heroku's
Read more

lex - Building a lexical Analyzer in Java -

-
I am currently learning lexical analysis in compiler design. To know how a lexical analyzer actually works, I am trying to build myself. I am planning to make it in Java The input to the lexical analyzer is a .tex file that is of the following format. \ section {script} {intro} \ section {Scope} arbitrary text \ section {relevance} uncontrolled text \ subdivision (profit) arbitrary text \ subsubsection {in real life} \ subdivision {Ingredient} \ end {document} The output of the lecture may be a table of contents with a page number in another file. 1 Introduction 1 1.1 Scope 1 1.2 Relevance 2 1.2.1 Benefits 2 1.2.1.1 Real life 2 1.2.2 loss 3 I hope that This problem is within the scope of Language Analysis Read the .exe file and check '\' and continue to check whether it is actually on search In the sectioning command is set to indicate whether or not a flag variable is the type of sectioning. I hope the above approach will work for the construction of Lexa
Read more

python - rename keys in a dictionary -

-
I want to change the names of keys in a dictionary which are ints, and I need them so that the inputs with key zero They can sort correctly. For example, my keys are like this: '1', '101', '11' and I may need them: '001', '101', '011' What is doing, but I know that there is a better way tmpDict = {} in the ADC for the old: t MpDict ['% 04d'% int (olden)] = addict [oldkey] New controlled = tmpDict You are going wrong about this if you want to draw entries on a line in a specific way, then you have to sort on the extraction. for sorted (D, key = int): print '% s:% R'% (k, d [k]) < / Div>
Read more