You can also use awk, which should be available on every linux/os x machine (sorry for the bad formatting, if you wish a can send you a nicely formatted version by email):

#!/bin/sh

awk '/^C /,/^C/ {lastc = $0; next}

{ if ( lastc != "") { print lastc; lastc = ""; }

print }' $1

Save the script as 'chkheadings' (or whatever name you like) and run it from a command line as:

sh chkheadings input_file

(or add execution permissions to 'chkheadings' and run it without sh).

Please let me know if you have problems.

You can also use awk, which should be available on every linux/os x machine (sorry for the bad formatting, if you wish a can send you a nicely formatted version by email):

#!/bin/sh

awk '/^C /,/^C/ {lastc = $0; next}

{ if ( lastc != "") { print lastc; lastc = ""; }

print }' $1

Save the script as 'chkheadings' (or whatever name you like) and run it from a command line as:

sh chkheadings input_file

(or add execution permissions to 'chkheadings' and run it without sh).

Please let me know if you have problems.

Both Sabino and Sven have written very terse code - though Unix folks (of which I include myself) often do so :-) I'm pulling out my perl and awk books to completely understand the solutions.

The awk script (Sabino), generates the output Muhammad is looking for.

Sven, I can't identify the problem, but running your code doesn't quite get there. Based on the test input and your code I get:

C 0001 Carbon [C]

D SAR001 methane [CH3]

D SAR002 ethane

D SAR003 propane

D SAR004 butane

D SAR005 pentane

C 0002 Hydrogen [H]

D SAR011 ozone

D SAR012 super oxide

C 0005 Sulphur [S]

D SAR013 Hydrogen Sulphide [H2S]

D SAR014 Sulphuric acid

Note that C 004 is excluded, and C 002 appears to have some D entries.

I don't know Perl but, from what I see (and from my awk script), I would guess the problem is that the variable $drop is never reset in the while cycle.

But I might be wrong...

i believe Sven meant $drop =~ /^C/;

Pedro, unfortunately, that doesn't fix the problem - it turns the code into an echo of existing input :-(

Without an explanation of what is happening, it's not easy (at least for me) to solve the problem. It is why I write more verbose code if I'm giving it to someone else.

Sven's code drops the second C in

C

C

D

rather than the first. The test for next happens after the first C has already been printed out. Hence:

C 0002 Hydrogen [H]

C 0003 Nitrogen [N]

C 0004 Oxygen [O]

D SAR011 ozone

D SAR012 super oxide

The lines for Nitrogen and Oxygen get dropped rather than the hydrogen.

Also consider any data beginning with :

C

C

D

drop is initially false, so the the first line will always be printed out.

I would probably do something like:

#!/usr/bin/perl -w

$previous = ;

while(){

if($previous =~ /^C/ && $_ =~ /^C/){

$previous = $_;

next;

}

print $previous;

$previous = $_;

}

which doesn't look very elegant, but does work. To run use:

cat filename | scriptname

Thanks Lars - that clears up what was happening.

Thank you Lars Jakt for your answer. It exactly did what I needed !

Thank you to Sven Bilke and Sabino Maggi for your valuable suggestions.

And thank you Jerrold Heyman and Pedro Reche for your useful comments.

I like the awk solution by Sabino. Here goes a perl counterpart that also works well.

#!/usr/bin/perl -w

while() {

if (/^C/){

$lastc = $_;

next;

}

if ($lastc ne '') {

print $lastc;

$lastc = '';

}

print $_;

}

Jerrold, you are absolutely right, all code should be fully explained to be useful. However, it was a quick solution...

Today I have uploaded here on RG a fully commented script, with detailed instructions for running it.

The pdf printout of the script, together with the dowloadable awk script and a couple of input/output test files is at the address:

https://www.researchgate.net/publication/259344360_chkheader.awk_An_awk_script_to_conditionally_delete_lines_in_a_KEGG_database_Brite_hierarchy_file

Data chkheader.awk: An awk script to conditionally delete lines i...

My 20c in Ruby :)

cu=false

pr=false

File.open("test.txt").each do | l |

if cu

pr=cu

end

cu=l

if pr

unless cu[/^C/] and pr[/^C/]

puts pr

end

end

end

if cu[/^D/]

puts cu

end

This can be done using grep:

$ cat input.txt | grep --no-group-separator -B 1 '^[^C]'

In other words, print any line that doesn't contain C at the start, as well as the previous line [but don't duplicate printed lines].

David: Great!

Perl equivalent one-liner with an END section:

$ cat input.txt | perl -ne '{print $last if(/^[^C]/ || $last =~ /^[^C]/); $last = $_} END {print $last if $last =~ /^[^C]/}'

You might like to have a look at www.commandlinefu.com it is a place to store and search for useful linux commands...

perl

tr/C\s+\d+\s+\w+\s+\[\w\]//i;

@Matthew Olson: That perl script looks like you're trying to just delete lines that start with C, which doesn't match the request (there's a pre-requirement for deleting those lines). Furthermore, the 'tr' operator is used to replace characters -- I think you meant 's' instead of 'tr'.

Except with tr, you're replacing all the symbols regardless of what follows, so *all* word characters on all lines, as well as all spaces. Even then, you need to add a /d flag to actually delete characters, otherwise you're just doing a counting operation (assuming this is a Perl script).

I still don't think your 'tr' string would work (and running it myself comes up with a syntax error). Have you tried running it on the example input file?

OK I can't get tr to work either in this instance, and my use of tr or s is by trial and error; so I set out to make this work for everyone. Here's the full script. Try it and let me know if it works for you:

use strict;

open (IN, "test.txt")|| die "can't open $!";

my $fileasstring;

while (){

chomp;

$fileasstring .= $_."\n";

}

while($fileasstring =~ m/C\s\d\d\d\d\s\S+\s\[\S+\]\nC/) {

$fileasstring =~ s/C\s\d\d\d\d\s\S+\s\[\S+\]\nC/C/;

}

open (OUT, ">testout.txt") || die "can't open $!";

print OUT $fileasstring;

Well, since you've put in Perl code and there are already two other Perl solutions, it seems reasonable for me to run through this code and look at how it is different from the previous solutions:

> my $fileasstring;

> while (){

> chomp;

> $fileasstring .= $_."\n";

> }

It's not particularly good practise to read an entire file into memory like this, particularly when there are easier ways to do it in an almost line-by-line fashion that works with pipes (like the other examples here, which store only the previous line and current line in memory).

> while($fileasstring =~ m/C\s\d\d\d\d\s\S+\s\[\S+\]\nC/) {

> $fileasstring =~ s/C\s\d\d\d\d\s\S+\s\[\S+\]\nC/C/;

> }

Okay, but you can make this more generic and use the /m flag. It looks like you're paying too much attention to the specific input file without dealing with the more generic solution that was asked for:

> while($fileasstring =~ /^C.*\nC/m) {

> $fileasstring =~ s/^C.*\nC/C/m;

> }

Alternatively, Perl has {n} for counts of matches, so \d{4} will match 4 digits.

For more information, see the Perl documentation on regular expressions:

http://perldoc.perl.org/perlre.html

...and then there's also no accounting for the last line starting with C, but things like that are usually fairly easy to work around by other means.

I don't feel comfortable with code that cannot be intuitively understood. Scala can be used as comfortable, not overly concise scripting-like language, is as powerful as Haskell and can also access all java libraries like biojava.

two-liner:

val a = scala.io.Source.fromFile("/home/ingvar/testfile").getLines.zipWithIndex.toArray

val result = a.filter{case (line, index) => if (line(0) == 'D') true else if(a(index+1)._1(0) == 'C') false else true}

more optimized tail-recursive lazy version for big files / memory-limitation:

val a = scala.io.Source.fromFile("/home/ingvar/testfile").getLines

import scala.annotation.tailrec

@tailrec def filterC(input:Iterator[String],

accu:List[String] = List.empty[String]):List[String] =

if(input.hasNext){

val line = input.next

(line(0), accu) match {

case ('C', Nil) => filterC(input, line :: Nil)

case ('C', x :: xs) if x.startsWith("C") => filterC(input, line :: xs) // disgarding x, the previous C

case ('C', x :: xs ) if x.startsWith("D") => filterC(input, line :: x :: xs)

case ('D', x :: xs) => filterC(input, line :: accu)

case (_,_) => throw new Exception("something wrong")

}

}

else accu

val result = filterC(a).reverse

(for running: In the scala REPL (~shell), write ":paste", paste the code below, then confirm with enter + ctrl-D).

The following one-liner should work for you:

cat testfile.txt |awk 'BEGIN{RS="C "} {print "C "$0} END{}'|tail -n +2|awk 1 ORS=';'|perl -p -E 's/C([^\;]+\;\;)//g'|sed 's/\;\;/\;/g'|sed 's/\;/\n/g'

To Sabino Maggi: grep can accept a file name as an argument, it is not needed to use cat and a pipe. The following is sufficient:

grep --no-group-separator -B1 '^[^C]' input.txt

In Linux you can use a grep command to do this:

grep -v -B 1 ^"C " input.txt | grep -v -e --

Note that the -v flag inverts the sense of the grep search, and -B1 prints the line before the match and the match line (and consolidates any repeat lines). So combining these flags prints all lines except the lines before a line that starts with "C ".

The second part of the piped command removes the -- separator

Here's the results of a test:

C 0001 Carbon [C]

D SAR001 methane [CH3]

D SAR002 ethane

D SAR003 propane

D SAR004 butane

D SAR005 pentane

C 0004 Oxygen [O]

D SAR011 ozone

D SAR012 super oxide

C 0005 Sulphur [S]

D SAR013 Hydrogen Sulphide [H2S]

D SAR014 Sulphuric acid

It is almost the same as Sabino Maggi's answer, nut int the former --no-group-separator option is used to suppress the group separator, thus the second grep is not needed.

perl -pe 's/^(C.+)\n/\1___/g' test.txt |perl -pe 's/^C.+___C/C/g'|perl -pe 's/___/\n/g'

To clarify for Richard (I am not a programmer BTW :-):

test.txt

C 0002 Hydrogen [H]

C 0003 Nitrogen [N]

C 0004 Oxygen [O]

D SAR011 ozone

perl -pe 's/^(C.+)\n/\1___/g' test.txt

C 0002 Hydrogen [H]___C 0003 Nitrogen [N]___C 0004 Oxygen [O]___D SAR011 ozone

(replaces the newline (\n) after each line starting with C, with 3 underscores, effectively concatenating those lines.)

|perl -pe 's/^C.+___C/C/g'

C 0004 Oxygen [O]___D SAR011 ozone

(takes the previous result and keeps only the last occurence of the C and following characters in the line)

|perl -pe 's/___/\n/g'

C 0004 Oxygen [O]

D SAR011 ozone

(then simply replaces the 3 underscores by a newline.)

May not be the most elegant way, but it works and it's short.

I hate scripts that cannot

- be understood easily

- that have no comments (here comment for non programmers :-).

Please take a look at this Python code (since I found Python, I dont use Perl anymore...).

# first open file for reading

fichin=open('/home/test_file', 'U') # unix comment if windows

#fichin=open('\C\tmp\test_file', 'U') # uncomment if window

tmp='' # to store the content of a "C line"

for li in fichin: # read one line at a time, => file can be >100 Go

if li[0]=='C': # line starts with a C

tmp=li

else: # line does not start with a C

if tmp!='': # previous line started with C, because tmp is not empty

print tmp.strip() #remove \n

tmp='' # empty tmp

fichin.close() # close file (not necessary, but better)

with 3 lines more => ouput in a file

fichout=open(xxx, 'w')

...

print tmp ==> fichout.write(tmp)

...

fichout.close()

FWIW, Python doesn't really work well for ResearchGate comments because there's no way to do precise indentation, and there's no way to represent code blocks in Python without indentation. Consider the following:

> if tmp!='': # previous line started with C, because tmp is not empty

> print tmp.strip() #remove \n

> tmp='' # empty tmp

> fichin.close() # close file (not necessary, but better)

While I can guess at where the if statement ends (after tmp=''), that's not something a naive programmer would be able to do, and may choose one of the two other incorrect indentation finishing points.

If you really want to include python code, I recommend using a symbol to represent indentation (e.g. '>'), or attaching the code rather than embedding it in the comments.

Regarding your code, where are you printing the current line? Shouldn't there be something like 'print li' in there? Have you run this code with the example input to verify that it produces the expected output?

Ignoring that, the code looks fine, but could be improved (if you want to do that) by removing the specific file names and allowing commsnd-line file options. That would mean it doesn't need changing every time you work on a different file. You can use the fileinput module for an easy way to do this:

http://stackoverflow.com/questions/715277/how-do-i-iterate-over-all-lines-of-files-passed-on-the-commandline-in-python