remove SVN headers (or include them full) in last commit of release

maniqui · 2012-07-03 22:49:58

Problem: (text taken from here, see last question)

Basically, on each TXP official release (distributed on .zip/.tar.gz files), the checksum for each file is calculated including an SVN header with some SVN-related info (HeadURL and LastChangedRevision) for that file.

For example:

/*
$HeadURL: https://textpattern.googlecode.com/svn/releases/4.4.1/source/index.php $
$LastChangedRevision: 3189 $
*/

But you will find that, even if you checkout an official tag/release (like 4.4.1), although this kind of SVN header is present on each file, it doesn’t have the proper info/values. Instead, it looks like this:

/*
$HeadURL$
$LastChangedRevision$
*/

In other words, the SVN header is almost empty, and, in consequence, the file itself is different from the “official” file (bundled on an official, downloadable release) for which the known checksum applies. Thus, you will inevitably get the “Some Textpattern files have been modified” in the Diagnostics tabs.

TL;DR: the official release (the .zip or .tar.gz file) of Textpattern doesn’t match exactly with the corresponding official commit (for TXP 4.4.1, it would be the commit r3576 in the graph below) in the SVN repo. This means ugly red lines in Diagnostics tab.
This mismatch… I think it could and should be avoided.

Proposal:

For those checking-out/testing/running from a stable release on the SVN repo (or better, a Git repo), this is an inconvenience.

I’d like to ask, if possible, for next release, to get this (useless?) SVN headers removed (or at least, included in full) in the release branch.

As an example, looking at the graph below (which corresponds to the release branch of TXP 4.4.1), the revision r3576 should have included the full SVN headers.

* r3577 - dev is on 4.4.1 now. 
| * r3576 - (4.4.1) Checksums for 4.4.1
| * r3576 - This is Textpattern CMS 4.4.1 stable
| * r3574 - Copying the 4.4.1 branch
|/  
* r3572 - Language files for 4.4.1

TL;DR (part 2): could the SVN headers be removed, please?

Thanks for reading.

Gocom · 2012-07-03 23:27:36

While I personally do not like the automatic processing SVN offers, or does, and I do not like tags, Textpattern does use tags correctly and I’m very fine them being there. Textpattern even actually uses those values, and seeing that information in files can be helpful even to some.

maniqui wrote:

But you will find that, even if you checkout an official tag/release (like 4.4.1), although this kind of SVN header is present on each file, it doesn’t have the proper info/values. Instead, it looks like this:

As it should actually. Those things you are seeing are keywords. SVN doesn’t expand keywords on checkout (as designed). So, you see them in their raw form.

If you need a release branch with keywords expanded, you can do so by using svn export command. Thing is, checkout is the wrong command for that.

I’d like to ask, if possible, for next release, to get this (useless?) SVN headers removed (or at least, included in full) in the release branch.

What do you mean by included in full? Keywords are handled by SVN when exporting. They are not committed and shouldn’t be committed with a branch. If you were to do so, at that point you could as well do copies instead of branching.

Anyhow, instead of removing the tags, if there needed to be a fix to the checksum deal, I would just disable checksum checks on SVN/git installs/checkouts. The checksums are valid for only stables anyways.

Since/when you are using SVN/git, the checksum are not even relevant. You can just use the repository to verify the file integrity. Git will wine if it sees something strange, and unlike bundled checksums, the repository verification is secure and safe no matter how the files were touched/modified.

Last edited by Gocom (2012-07-03 23:33:57)

maniqui · 2012-07-04 00:15:59

Gocom, I’m an SVN illiterate, thanks for the info.

Textpattern even actually uses those values (…)

Good to know.

Anyhow, instead of removing the tags, if there needed to be a fix to the checksum deal, I would just disable checksum checks on SVN/git installs/checkouts. The checksums are valid for only stables anyways.

I understand.
But then, stable Textpattern 4.4.1 coming from zip/tar.gz is different (include SVN headers) from the corresponding commit for stable Textpattern 4.4.1 (doesn’t include SVN headers). In other words, the only difference are the SVN headers.

Of course, I’m considering that “stable” doesn’t (and shouldn’t) mean “just the one bundlied in the zip/tar.gz package”.
The 4.4.1 version in the SVN repo is equally functional (and equally stable) to the one on the zip/tar.gz package.

wet · 2012-07-04 02:37:12

This is the script we use to build the release archives. You can do the same from every tagged branch.

#!/bin/sh

if [ $# -lt 1 ]; then
        echo 1>&2 Usage: $0 '<version> [svn-repos]'
        exit 127
fi

VER=$1
TMPDIR=`mktemp -d`
REPOS=http://textpattern.googlecode.com/svn/development/4.x/

if [ $# -eq 2 ]; then
        REPOS=$2
fi

svn export $REPOS $TMPDIR/textpattern-$VER
tar cvf - -C $TMPDIR textpattern-$VER | gzip -c > textpattern-$VER.tar.gz
OLDDIR=`pwd`
cd $TMPDIR
zip -r $OLDDIR/textpattern-$VER.zip textpattern-$VER
cd $OLDDIR

Problem solved.

maniqui · 2012-07-05 11:50:47

I’d have preferred to have the contents of a particular release to be same, without differing if I get them from a zip file or checking them out from a particular tag on a repository.

Anyway, thanks Gocom & wet for your comments.

Gocom · 2012-07-05 12:22:56

maniqui wrote:

I’d have preferred to have the contents of a particular release to be same, without differing if I get them from a zip file or checking them out

You can, with svn export. svn checkout is the wrong tool if that is the purpose. Checking out always gives you a working copy.

Please note you don’t need checkouts for release branches. Branches are final and won’t be updated.

Last edited by Gocom (2012-07-05 12:24:15)

maniqui · 2012-07-05 12:39:44

I’m using “checking out” more in a Git-sense: switching/transforming your working directory to match a certain commit/branch/tag. Also, yes, I use tag in the sense that the git mirror (not the official one, but the one I made) uses a tag (eg. “4.4.1”) on the tip of each release branch.

That said, a question that may help explain my not-so-OCD requests:
Which is Textpattern 4.4.1?

r3575. This is Textpattern CMS 4.4.1 stable.. This?

r3576. Checksums for 4.4.1.. Or this one maybe?

r3577. dev is on 4.4.1 now.. Or maybe this one?

textpattern-4.4.1.zip. Or just this one?

I’d love the repo to have at least one commit where the files (at least, for files that are checksummed) match the files on the zip package.
But it’s ok if not possible, I can live without that. I now understand there is a build/deploy process that couldn’t make this possible.

Gocom wrote:

Textpattern even actually uses those values (…)

Back to that, could you shed some light about how TXP uses those values? Maybe for checking for new updates? Or anything more substantial?

Thanks again.

Last edited by maniqui (2012-07-05 12:40:57)

Gocom · 2012-07-05 13:49:12

maniqui wrote:

Which is Textpattern 4.4.1?

http://textpattern.googlecode.com/svn/releases/4.4.1

Is v4.4.1. The last commit to that branch is the exact the source you find from the zip. You get a release by exporting a branch, or local copy of the repository by a checkout.

I’d love the repo to have at least one commit where the files (at least, for files that are checksummed) match the files on the zip package.

Yes the repository contains all the raw source files, and all of those files do match the checksum. I see no messages on my Diagnostics panel at all. You don’t need to do any building or anything like that. You can get v4.4.1 by running:

svn export http://textpattern.googlecode.com/svn/releases/4.4.1/source

Or checkout:

svn checkout http://textpattern.googlecode.com/svn/releases/4.4.1/source

Both give you 100% match. First a source, second a working copy. In export SVN properties are parsed and .svn directories are stripped (as expected if you want the exact copy of the zip).

Since Git isn’t SVN, Git doesn’t handle SVN related features as keywords or other file properties. On git you will lose Textpattern’s branches, keywords and other SVN stuff. These are either stripped out, or you find the raw properties committed to the repository (i.e. keywords).

But it’s ok if not possible, I can live without that. I now understand there is a build/deploy process that couldn’t make this possible.

Can you please explain what build process you are referring to? There is no building involved, other than that Robert generates checksums and commits them to the repository for us. :o I’m seriously confused.

Are we talking about the keywords? The raw keywords are in the git repo since Git isn’t SVN. If Textpattern team uses SVN related features, I’m absolutely fine with that. I do not mind. Anyways, repositories are used to get the source. Both on git and SVN you get the same source when doing a checkout. Git just can’t handle tasks that SVN would. I use the git repo and I’m fine with what it offers.

Back to that, could you shed some light about how TXP uses those values? Maybe for checking for new updates? Or anything more substantial?

Dunno, I’m not Textpattern developer. To me they tell which revision the file is. Can help with development process since you clearly know what the file is.

Last edited by Gocom (2012-07-05 13:51:07)

maniqui · 2012-07-05 14:15:33

Gocom wrote:

http://textpattern.googlecode.com/svn/releases/4.4.1
Is v4.4.1. The last commit to that branch is the exact the source you find from the zip. You get a release by exporting a branch, or local copy of the repository by a checkout.

Yes the repository contains all the raw source files, and all of those files do match the checksum.

This file
textpattern.googlecode.com/svn/releases/4.4.1/source/index.php
is different from the one in the zip file.

Yes, the only difference is this:

<?php
/*
$HeadURL$
$LastChangedRevision$
*/

!=

<?php
/*
$HeadURL: https://textpattern.googlecode.com/svn/releases/4.4.1/source/index.php $
$LastChangedRevision: 3189 $
*/

That alone will make the checksums fail.

That’s what I’ve tried to explain so far, so, sorry if I’m being too repetitive.

Since Git isn’t SVN, Git doesn’t handle SVN related features as keywords or other file properties. On git you will lose Textpattern’s branches, keywords and other SVN stuff.

That’s why I tried to solve by creating this mirror of Textpattern SVN repo on Github, which differs from this (now official) one.
I’ve just checked again and my mirror has some commits (corresponding to r3576, r3576, r3574, see graph in OP on this thread) that aren’t even present in the official mirror! There are aswell other commits (from previous releases) that aren’t present in the official mirror but that are present on mine.

The history in my mirror is/looks more similar to the one on the Textpattern SVN repo: branches (releases) for each release clearly defined & no commits missing. If interested, I’ve tried to explain this better and with more detail in the repo wiki.

Can you please explain what build process you are referring to? There is no building involved, other than that Robert generates checksums and commits them to the repository for us. :o I’m seriously confused.

Sorry, I may have misused a word, once again. By “build” I mean a process where the source files are modified/transformed/compiled into another file, similar but not identical. In this case, the “build” would be to add all that SVN-related info. That info is not in the source file, but it’s there in the zip package, and I assume by your clear explanations, the SVN-related info gets added there if you do an SVN export/checkout. That’s what I meant with “build”.

Last edited by maniqui (2012-07-05 14:20:05)

Gocom · 2012-07-05 14:46:05

maniqui wrote:

I’ve just checked again and my mirror has some commits (corresponding to r3576, r3576, r3574, see graph in OP on this thread) that aren’t even present in the official mirror!

The official mirror mirrors the dev branch (as should, you can’t mirror multiple sources w/ messing shit up). Those commit are to different branch, aren’t they? Wait, are you saying that your mirror has commits that are to different branch then, merging commits from different sources? If that’s the case, then the history on your mirror doesn’t match the one of dev branch.

If you want to do mirrors as true to the SVN as possible, those tags should be branches and those missing commits should go to those branches. Not to the master.

Yes, the only difference is this: […] That alone will make the checksums fail. […] That’s what I’ve tried to explain so far, so, sorry if I’m being too repetitive.

Thing is, unlike Git, SVN isn’t distributed system. What you are looking is the file SVN stores on the server. It relies on the server, and it stores raw and “substitution” files on the server. Those files are the same, and there is no building involved. When I run:

svn checkout http://textpattern.googlecode.com/svn/releases/4.4.1/source

I get the source. It’s 100% match to the ones on the ZIP, and checksums are identical. The difference include that there is .svn data present in the checkout directory. If I want the 1on1 Zip, I would use export.

Thing is, SVN works that way. When you do checkout or export, SVN handles it raw properties. For instance, SVN doesn’t store keywords, file properties or special files (as symlinks) on the server in their true format. It creates and sets those locally. You are looking at the raw file stored on the server… SVN unfortunately isn’t distributed, there is no clones.

The files git has and Github host are exact copy of what the SVN server stores. That’s the only way to keep the history the same on both systems. But since Git doesn’t handle SVN’s features, when you clone the git repository you get what was on the SVN server. This is due to git being distributed system where everything is a clone and an Independent instance of the repository.

Last edited by Gocom (2012-07-05 14:50:13)

maniqui · 2012-07-05 15:06:35

The official mirror of the dev branch.

I will be a bit picky, sorry.
GitHub repo states: “An official mirror of the Textpattern SVN codebase”

Those commit are to different branch, aren’t they? Wait, are you saying that your mirror has commits that are to different branch then, merging commits from different sources? If that’s the case, then the history on your mirror doesn’t match the one of dev branch.

The history of my mirror matches the one on dev branch and also includes the branches.

This is a bit of it…¹

(...)
* 9141c74 - Really escape banned IP addresses and names in ban list. (1 year ago) <r.wetzlmayr>
* 6c92860 - Escape banned IP addresses and names in ban list. Thanks, Sasha Zivojinovic. (1 year ago) <r.wetzlmayr>
* bd680de - dev is on 4.4.1 now. (1 year, 1 month ago) <r.wetzlmayr>
| * 9ca3e45 - (4.4.1) Checksums for 4.4.1 (1 year, 1 month ago) <r.wetzlmayr>
| * 65c35d2 - This is Textpattern CMS 4.4.1 stable. (1 year, 1 month ago) <r.wetzlmayr>
| * 682a4f7 - Copying the 4.4.1 branch (1 year, 1 month ago) <r.wetzlmayr>
|/  
* ee226c4 - Language files for 4.4.1 (1 year, 1 month ago) <r.wetzlmayr>
* 77117e1 - Destroy user's nonce upon logout. Thanks, Gocom. (1 year, 1 month ago) <r.wetzlmayr>
* 5a7827f - Rogue semicolon removal; (1 year, 1 month ago) <stefdawson>
(...)
* 678d184 - Update phpass to version 0.4 /genuine. Fixes issue #107. Thanks, Solar Designer. (1 year, 3 months ago) <r.wetzlmayr>
* 8098606 - dev is on 4.4.0 now. (1 year, 3 months ago) <r.wetzlmayr>
* 6817e9c - Fix checksum file list for trunk. (1 year, 3 months ago) <r.wetzlmayr>
| * f07d979 - (4.4.0) Checksums for 4.4.0 (1 year, 3 months ago) <r.wetzlmayr>
| * b373460 - Fix checksum file list for 4.4.0. (1 year, 3 months ago) <r.wetzlmayr>
| * d4e0494 - This is Textpattern CMS 4.4.0 stable. (1 year, 3 months ago) <r.wetzlmayr>
| * c7c4b23 - Copying the 4.4.0 branch (1 year, 3 months ago) <r.wetzlmayr>
|/  
* 6727868 - Writing HISTORY.txt (1 year, 3 months ago) <r.wetzlmayr>
* 1c49e3a - Language files for 4.4.0 (1 year, 3 months ago) <r.wetzlmayr>
* 0e3b963 - Documentation: Taghandler functions must not use upper case letters. Side effect of get_defined_functions(). PHP is fun. (1 year, 3 months ago) <r.wetzlmayr>

As you can see, the history of dev branch not only matches exactly the one on SVN (or in official mirror, for the case) but it also has the branches clearly identified.

I’ll look if git svn has some magic that, combined with git checkout some_branch could help to recreate the files that would match the checksum. That is, with the correct SVN headers.
I think my Git mirror has all the svn metadata stored, although I’m not totally sure about that.

Thanks, Gocom!

¹ Output of git log --color --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(blue)<%an>%Creset' --abbrev-commit.

Gocom · 2012-07-05 15:16:29

maniqui wrote:

The history of my mirror matches the one on dev branch and also includes the branches.

Your mirror has tags, not branches, right? All your mirrored commit are in the same branch, right? When you clone the repository and pull a master you get commits for all branches and all tags, right?

I’ll look if git svn has some magic that, combined with git checkout some_branch could help to recreate the files that would match the checksum. That is, with the correct SVN headers.

The keywords are correct as they are now. If you commit the headers to the repository with keywords substituted, then repositories’ history will be different making the git mirror — well — useless. Each commit would include a header changes.

Last edited by Gocom (2012-07-05 15:19:53)

Textpattern CMS

Textpattern CMS support forum

#1 2012-07-03 22:49:58

remove SVN headers (or include them full) in last commit of release

#2 2012-07-03 23:27:36

Re: remove SVN headers (or include them full) in last commit of release

#3 2012-07-04 00:15:59

Re: remove SVN headers (or include them full) in last commit of release

#4 2012-07-04 02:37:12

Re: remove SVN headers (or include them full) in last commit of release

#5 2012-07-05 11:50:47

Re: remove SVN headers (or include them full) in last commit of release

#6 2012-07-05 12:22:56

Re: remove SVN headers (or include them full) in last commit of release

#7 2012-07-05 12:39:44

Re: remove SVN headers (or include them full) in last commit of release

#8 2012-07-05 13:49:12

Re: remove SVN headers (or include them full) in last commit of release

#9 2012-07-05 14:15:33

Re: remove SVN headers (or include them full) in last commit of release

#10 2012-07-05 14:46:05

Re: remove SVN headers (or include them full) in last commit of release

#11 2012-07-05 15:06:35

Re: remove SVN headers (or include them full) in last commit of release

#12 2012-07-05 15:16:29

Re: remove SVN headers (or include them full) in last commit of release

Board footer