hierarchically cluster in R?

Once you have your data cleaned (i.e. removed the first column), this really just requires three lines of code:

Clean data (assign row names from first column, then remove first column):

dat <- mtfx.in
rownames(dat) <- dat[, 1]
dat <- dat[, -1]

Cluster and reorder:

row.order <- hclust(dist(dat))$order
col.order <- hclust(dist(t(dat)))$order

dat[row.order, col.order]

Results:

     col5 col4 col1 col3 col2 col6
row6    0    0    0    0    3    3
row5    0    0    0    0    3    3
row1    0    0    0    0    3    3
row3    0    0    0    0    3    3
row2    6    6    6    6    6    6
row4    6    6    6    6    6    6