Find the oldest file in something in the following way (change the directory name accordingly)
var oldestFile = FastDirectoryEnumerator.EnumerateFiles(@"c:\windows\system32")
.OrderBy(f => f.CreationTime).First();
or
var oldestFile2 = new DirectoryInfo(@"c:\windows\system32").GetFiles()
.OrderBy(f => f.CreationTime).First();
Now delete the file using the method Amit suggested i.e. DeleteFile (you may need to add directory name to make it absolute path)