hi

this link is useful for your ask:

http://www.industry.siemens.com/drives/global/en/motor/pages/default.aspx

Hi Seyed,

I fail to see the relevance of the link you provide. Can you please elaborate?

Thanks

For sites suitable for caching, you might want too look into the http header, more specifically the (optional) Etag response header:

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/ETag

There's also a 'last modified' response header, but this one is less accurate. I find the scanning however the most convenient and trustworthy as it also deals with sites that are not good candidates for caching. Once you've automated that, there's not soo much too it.

The institutional websites update based on their programmes and projects. It can be a criteria. 

A web site is usually composed of severl (and more likely, MANY) "pages".  The "last modified" and ETAGs, etc., noted above apply only to the asset to which they are "attached".

Each web page/script/css/image, etc. on the web site can have it's own "last update" date, and ETAG, and expiration, etc.  So, unless the webmaster (or the platform/framework/etc. that they are using) actually "stamps" the time of the last change anywhere on the web site at some central location (perhaps on a page footer on the "home" page, for exaplel, the only way to tell that last time the site was updated is to browse all of the web asset directories and look at the "last modified" date for each and every asset - and determine the asset that was most recently changed..

To find the frequency of updating for a website, you'd have to accumulate the "last update" date/time for each asset, and note the "most recent" change to any asset at whatever granularity you choose (a website CAN be updated more than once a day - in which case, depending upon the granularity you desire, may require performing the "last update" check process several times a day).

Also, some CMS systems maintain the content (which is the data that changes) in a database - the actual "web site" itself doesn';t change (very often - mostly only when a new release of the CMS is installed) - all changes tot he content are to database records -

You can tray services like https://visualping.io/, which is monitoring any change on the web page and then gives you the change and the time when it was detected. The service is free for the smal number of pages.

Iris Steenhout, can please you elaborate a bit on "website suitable for caching?"

Aren't cached pages "snapshots" of a website in a given point in time? 

Thank you,

MM

The archive (wayback machine) is only a mostly random snapshot of a website that has been changed sometime since the previous snapshot. There may have been several or many changes since that previous snapshot - every change to the website is not archived there (it would be fairly difficult to check every page of every web site every day in order to track all changes that have occurred to all websites...).  So looking at wayback will not provide enough data for you to determine the frequency of changes to any given site - it may have been changed numerous times in between wayback snapshots..

Web sites can be changed via FTP, or via WEBDAV, or via CMS systems interfaces, or by directly saving pages from an editor (assuming that you can edit on a machine that has "write" access to the web directories), etc.. 

It would be convenient if all websites would  auto-update a footer or something whenever any page is changed on the site - or if a standard existed to encourage webmasters to do so manually, etc.,  But there's not any such standard, and many sites don't even post a "last updated" message anywhere (though some do...). This leaves only the operating system file timestamp for individual files, or a database record change date for CMS systems that keep "pages" in a database (assuming that the database even has a "last changed" field for each record)

ETAGs, and LAST MODIFIED, etc., are pertinent to a specific asset (web page, or graphic image, etc.), and in order to use them to assess the frequency of updates to the site, you'd have to interrogate every asset of the site every day (every page, every image, every javascript program, etc.) and determine the most recently updated time in order to determine the last time anything was changed on the site.  You would have to do this every day in order to keep track of frequency of changes to the site.

If the site belongs to you, you may be able to access the web directories directly, and use the OS timestamps to determine this.  Or you may have access to update logs that would provide the data for detecting changes, etc.  But if this is for a site not belonging to you, your only recourse to accurately determine frequency of change is to walk every asset of every site regularly and detect any changes since the previous walk.  Depending upon the"granularity" that you require for your project, you could do this daily, weekly, monthly, etc. (depending upon if your needs)

Hi Steve,

Thank you, your input is really interesting. My concern is not so much about frequency of updates but rather the last date a site was actually updated on. Often in their desire to be "evergreen", content authors or website managers avoid placing dates in their blogs or other uploaded material. Some sites may also be abandoned or updated selectively at unstable rates. Up until now the Cntrl+U View Source function works very indirectly for someone who does not read code to understand what an indication of a specific date means. I was hoping there is a better and more reliable way of obtaining something that should be available in the first place. A user should not have to chase this information front and back, it's a matter of "website ethics" in my humble opinion.

MM

That would be nice... but would require something along the lines of a "web manager" application through which ALL updates to the site are made, and a catalog of some kind where the changed dates would be saved by said "manager".

This would also require all websites to use said "manager" software (or compatible alternatives) to standardize where the "last changed date" is kept, and its format, etc..

However, web sites can currently be updated by several means, and there is no centralized controller through which updates are required to be made.

You can update a website via FTP, WEBDAV, "direct entry" via CMS systems, physically copying/moving assets into the site (copy/paste, cp/mv, drag 'n drop,etc.). See, for example, http://openskywebdesign.com/howtoupdate.html .

Also, you'd have to agree on what actually constitutes an update. For example, would upgrading to a new version of a JavaScript framework constitute an update? (all text, images, and content remain exactly the same, but you change the version of JQUERY that you use on the site in order to remediate some security hole in the old version, for example) .

The Archive site checks sites for changes sporadically. The site may have actually been updated at time A, B, C, and D (each time some interval later than the previous, and not necessarily regularly spaced), but if the Archive last checked for changes between times B and C, (that is, happened to check sometime after update B, but before update C, and has not yet gotten around to check that site again), it will have a last updated copy of the site as it was at time B, even though two subsequent updates have been made. That's why the frequency of the updates versus frequency of checking matter (and the only reason that frequency was mentioned in above)

Hope this helps - if you find a solution, I'd definitely be interested...

Hi Steve,

Excellent input -- again. Thank you.

It's the content updates that worry me: in a research context it is imperative that one knows when an article or blog was published as it often defines the relevance of the material to latest developments. The researcher cannot speculate on the time the article was written simply because the author decided not to reveal this information. So actually my concern is about front stage material--the interface with the public-- and not JavaScript or other background software updates. Comments may also be non-relevant as they may be submitted years after an article was published.

I am now checking the link you provided in an effort to understand how these updates happen on the maintenance end. Hoping to isolate the CMS functions to understand how these pages can be edited and how/if one can access information without being the website manager.

Grateful!

MM

Just open the page in question, then type

javascript:alert(document.lastModified)

into your address bar.

Cheers!

That may work with some browsers, but most (current) browsers strip off the "javascript:" as part of the input validation - which is for security reasons. You'll likely have to re-enter it to get it to work. But better than nothing!

Lennart and Steve,

This is really informative, thank you both!

MM

@Steve: That's why I wrote "type" instead of "paste" ;)

It brings up today's date rather than the last update - any further instructions would be appreciated.

By the way, it might not work with more modern browsers, but IE still appears to allow it.

@Debra: might be an active page that just actually was updated the same day..?