Use Python to write to a .txt file column by column

How do you append a new column of data to an existing .txt file? Basically I'm generating 5 dictionaries and each time I generate one I want to write the values to a master text file in a new column. I'll display my code even though you won't be able to run it because it calls another program I wrote:

# Import personal module import graphGenerator as gg # Open file for writing data to case=open(r'J:\FOIL\mediansandmeans.txt','w') # Run code for i in range(5): # create a graph using NetworkX and a code I wrote to read in an edgelist from a txt file G=gg.graph_creator(i+1) # calculate degree of all nodes using NetworkX--returns a dictionary d=nx.degree(G,weighted=True) # print dictionary values to text file for j in d.keys(): case.write('%s\n' % d[j]) 

Why don't you use a database for this? If you want to stick with files, use pandas or numpy depending on what kind of data you are dealing with; otherwise you'll just have to read the entire file, add your column and then write the file again.

You may also want to consider building up the CSV in transposed form, so instead of adding a new column each time, you're just adding a new row (which you can do by appending to the file in-place). If necessary, transpose it when you're done, which means rewriting the file once, but that's better than N times.

Or, alternatively, keep each column in a separate file, and then merge them into one CSV when you're done.

4 Answers 4

Text files are stored sequentially; line two begins where line one ends. You can modify material in the middle, but to add even one character (or to delete even one), you need to read everything that follows and write it again in its new offset in the file. In other words, you must read and write out the whole file, or use a different storage model (e.g., a database), as others suggested.

If you really had to add information to a file column-wise, you could do it by writing out fixed-length lines, padded with spaces; you then seek through the file and overwrite some of the spaces with new data. I won't provide code because it's a terrible approach: fixed-length records went out with the 1970s. And I really don't think it's necessary or appropriate in your case.

Looking at your code, I don't think you need to be adding columns to a file. I think the best solution would actually be to collect the values in a two-dimensional array, so you can write them out all at once, in the desired format, when you're done. Unless you have gigabytes worth of points, there's no reason to write them out one column at a time.

Edit: Since you like the array idea, here's how to create it and write it out easily:

from collections import defaultdict degrees = defaultdict(list) for i in range(5): G=gg.graph_creator(i+1) d=nx.degree(G,weighted=True) for j in d.keys(): degrees[j].append(d[j]) for k in sorted(degrees.keys()): case.write("%s: %s\n" % (k, "\t".join(degrees[k]))) 

The "two dimensional array" is actually a dictionary of lists, in keeping with your version. (I take it that all returned dictionaries have exactly the same keys.) The code uses two handy python features: The defaultdict class saves you the trouble of creating each array row explicitly when recording the first column. And the output code joins the five values into a single tab-separated string for output.

Note also that unless you sort the keys of a dictionary, you'll get them in arbitary order-- usually not what you want on output.