Development: 4Suite/Ft/Lib/Uri.py

Minor optimization

Merged from maintenance branch (hopefully for the last time)

Work around bug in setuptools' sandboxing for easy_install.

Wrap local files the same as urllib to make the return value consistent

Added support for data URLs
Added support for custom pep302 psuedo-URL

Enhancements to make Ft.Lib.Uri.Relativize more robust when dealing with URIs having mid-path empty segments.  Also includes additional test cases covering such cases.

Switched to using urllib2.urlopen for retrieving URI content.

Make `Relativize` return relative references in a significantly larger set of cases, and allow it to still return restricted relative references with the `subPathOnly` argument.

Add `Relativize` to __all__ so that it will be incorporated into the autodox, and make the docstring more reST-friendly.

Implementation and testing for Ft.Lib.Uri.Relativize, a method to produce context-sensitive relative references from absolute URIs.  Also included is one minor doc fix.

removed 2 unused productions from URI grammar in regex setup;
in UriToOsPath, added encoding support that was mentioned in doc string but not implemented.

Fixes for emacs python mode syntax highlighting

OPTIMIZATION: prevent unnecessary function call overhead

typo fix for prev checkin

added comment explaining an implementation detail that apparently misleads some people to think,
despite an abundance of API documentation to the contrary, that Latin-1 based interpretation of
percent-encoded octets has been hard-coded into the PercentDecode() implementation

propagate OsPathToUri and UriToOsPath attemptAbsolute changes begun
by Uche, so repo inits will work again

Python unicode literals are stored in bytecode as UTF-8. Prior to Py 2.3,
the UTF-8 codec would raise an exception if it found an unpaired surrogate,
thus causing an error when a compiled module is imported, if that module
contains a unicode literal like "u'\udc00'" by itself. MvL explained the
situation and offered a workaround on python-list [1], and I have implemented
a variation of it here.

[1] http://mail.python.org/pipermail/python-list/2005-March/269948.html

Make attemptAbsolute the default in Ft.Lib.Uri.OsPathToUri, etc.  So that FileUrl can be deprecated
Fix test_meta_document.py and bug it uncovered

PercentEncode() now uses a character iterator that is UCS-2/UCS-4 Python build-agnostic

added methods to UriDict to support 'in' operator

In MakeUrllibSafe(), raise helpful "IDNA unsupported" exception for Python 2.2;
Misc doc updates, including references to rfc2396bis -> RFC 3986;
1/0 -> True/False and other misc cleanup;
Added RFC 2396/2732 definitions of reserved, unreserved characters in comments for future ref/compatibility enhancements

added SplitFragment()

Add new utility function Ft.Lib.Uri.FileUrl
RELAX NG framework improvements (and implement WXS ID and IDREF types)

citation of rfc2396bis "corrected"

corrections prev commit

Updates based on submitted patches from Raphael Bossek (Debian/4Suite maintainer)

* Move specialized resolvers to their own module
* Make first stab at fixing horrid InstallUtil dependency logic
* Ensure server core code uses a repository resolver

comment update

minor exception fixes

fixed botched indent that was affecting local file resolution in FtUriResolver

1. removed use of types module
2. replaced NormalizeEmptyComponents() with NormalizePathSegments() & NormalizePathSegmentsInUri() to reflect changes in final draft of RFC 2396bis
3. added StripFragment()
4. made UriResolverBase be strict and general-purpose, and created a new subclass,
FtUriResolver, that adds in the base URI lenience that we depend on in 4Suite.
5. in what is now FtUriResolver, normalize() was only checking that the base URI
scheme was supported, when in fact after resolving the reference to absolute form,
it is possible that the reference's scheme would prevail and not be checked. This
has been fixed. Various other spaghetti code is a little cleaner now.
6. moved FTSS_URI_SCHEME into Ft.Server forever, hopefully
7. noted more urllib bugs in comments
8. Absolutize() doc string adjusted
9. MakeUrlLibSafe() and UrlOpen() now ensure that whatever is passed to urllib
is str, not unicode, since urllib has many places where it cannot properly handle
unicode

export WINDOWS_SLASH_COMPAT

crufty comment cleanup

1. New functions to implement RFC 3151 (public ID <-> URN conversion) added.
2. PercentEncode() now supports '+' -> ' ' conversion.
3. PercentEncode() now supports customization of reserved set.
4. PercentEncode() now supports CRLF conversions.
5. Absolutize() docs section on same-doc refs corrected.
6. host name pattern used in MakeUrllibSafe() updated (affects very long hostnames)

OsPathToUri on posix was constructing 'file:/abspath' URIs rather than
'file:///abspath' when attemptAbsolute was set and the original path
was not absolute.

keeping up with RFC 2396bis:
255-char limit in reg-name is now a 'should', and a few productions were renamed

rather than use UserDict, subclass dict, which is faster (Py 2.2+ only)

urllib.urlopen() normally ignores HTTP error responses that
FancyURLopener doesn't recognize, e.g. 404 (!).
We will attempt to work around this so that an exception will be raised
in such cases.
Also updated some doc strings.

use isinstance(foo, types.bar) rather than type(foo) == types.bar

MakeUrllibSafe() now supports IDNs for Python 2.3 up;
also moved Windows ':' -> '|' path translation into MakeUrllibSafe()
and made it only apply that translation if the scheme is 'file'

minor rfc 2396bis grammar update

in UrlOpen, remove fragment from URL before dereferencing

date of RFC 2396bis draft updated

adjusted RemoveDotSegments to match RFC 2396bis draft 05

keyword args in exceptions fixed up;
made handling of forward slashes in windows paths configurable;
removed extraneous module name in resolver base

During a repo init, external entity / XInclude dependencies are checked in XML files by parsing the XML. If the reader can't successfully read a dependency, a UriException, in which the dependency's URI *may* be found, is raised. This URI data was not being stored in the exception in an ideal manner, and thus wasn't being accessed very intelligently, either. Also, the XML was not being read reliably, since in some cases (e.g. ExtendedCommand instances) it needed to come from _getContent(), not from .contentUri, a problem that was masked until a recent change to stop ignoring unexpected parsing exceptions during dependency checkins.This is all now fixed, with various checkins.

added UrlOpen to replace urllib.urlopen();
fixed __all__

beefed up OsPathToUri code comments about the temporary hack for compatibility with previous versions of the function.
urllib had a 'convenience' feature of not differentiating between the directionality of slashes in paths on Windows, which we exploit in our code. Until we can clean up those cases, we'll just leave in a workaround here.

__all__ added;
order of module contents rearranged a bit;
various doc string and comment updates;
names of module-level constants are now consistently uppercase;
new function PercentEncode supercedes urllib.quote;
new function PercentDecode supercedes urllib.unquote;
new function SplitAuthority parses an authority component;
new function MakeUrllibSafe facilitates passing good URIs to urllib;
various new URI normalization functions implement RFC 2396bis sec. 6;
normalization functions incorporated into UriDict;
function _eliminateDotSegments renamed to RemoveDotSegments;
class BaseUriResolver renamed to UriResolverBase;
function _getScheme renamed to GetScheme, implementation improved;
SchemeRegistryResolver raises proper UriException now;
regex setup functions wait to set globals until done (thread safety);
on 4Suite's supported platforms (nt and posix),
- function OSPathToUri is no longer dependent on urllib.pathname2url
- function UriToOsPath is no longer dependent on urllib.url2pathname

updated comments re: urllib.basejoin and urlparse.urljoin issues

updated regex to match grammar in latest RFC 2396bis draft

1. Documentation improved for OsPathToUri(), UriToOsPath(), BaseJoin()
2. OsPathToUri() now collapses dot segments if the path is absolute, so that
   the URI can be safely used as a base URI by Absolutize()
3. OsPathToUri() now better handles NetBIOS paths on Windows
4. UriToOsPath() now better identifies NetBIOS paths on Windows
5. UriToOsPath() now better identifies the drivespec on Windows
6. UriToOsPath() now discards queries, not just fragments
7. BaseJoin() no longer relies on the buggy & outdated urllib.basejoin() for
   path merging; it now uses the same algorithm as Absolutize(), which is the
   RFC 2396bis algorithm.  The URI working group's position is that this is
   what they intended all along.
8. BaseJoin()'s operation and role clarified

updated merge routine in Absolutize() to match current spec

Absolutize() should never return the base URI's fragment, even when the relative reference is empty string

doc string updates and IPv4address fix in validation regex

minor updates to the validation regex to keep up with daily changes to RFC 2396bis

updated validation regex to match Feb 2004 draft of RFC 2396bis (even simpler syntax)

slice notation is slightly faster than startswith() and endswith() calls

Lenient URI syntax validation patterns were removed, resulting in an API
change: MatchesUriRefSyntax() and MatchesUriSyntax() no longer take an
optional 'strict'/2nd argument.

Strict URI syntax validation patterns and various functions were updated in
accordance with RFC 2396bis (the new version of RFC 2396, which is still an
Internet-Draft). Functions affected are:

SplitUriRef()
  Now returns a 5-tuple, not a 6-tuple, reflecting
  the elimination of a formal distinction of an "opaque part"
  (it is now just another kind of path component).

Absolutize()
  Now always returns a URI string (never None), and takes 2 arguments,
  not 3, reflecting a change in the way same-document references are
  determined. Recognition of same-document references is shifted back
  to the caller. The caller also now bears the burden of ensuring that
  the base URI is one that uses hierarchical path components (so you must
  not, for example, pass a URN as a base URI, because this will no longer
  result in an exception, and results will perhaps not be what you want).
  For predictable operation, the base URI should also not contain '.' or
  '..' path segments.

_eliminateBackRefs()
  Absolutize() support function replaced now by the more correct
  _eliminateDotSegments()

All named groups were removed from the regular expressions for strict URI
syntax validation.

Various doc strings and comments were updated.

The unused BaseUriResolver.splitFragment() method was removed.

Error messages pertaining to document URIs and non-hierarchical base URIs
were removed.

elimination of '..' backrefs in a path that contains '//' was not working correctly

SchemeRegistryResolver documentation clarification

Merged with HEAD to keep it up to date

Removed extraneous import

unhardcode list of default supported uri schemes, add tests for schemeregistryresolver, fix test_uri breakage

Add support for multi-lne queries and variable defs in Versa query files.  e.g.
QUERY all()
  - rdfs:label -> *
(subsequent lines part of the same query must be indented)
Fix many bugs in Versa query file code
Rename InputSource._clone to InputSource.clone (since we insist on using it as a public method, anyway)
Minimize use of private InputSource functions a bit
Allow FtssInputSources to delegate to non-ftss schemes if certain hints are specified (esp hint="XSLT DOCUMENT FUNCTION")
Use URI resolution hints more consistently
Use uppercase for global functons in Uri module
Fix bug whereas if XSLT A imported XSLT B and the latter used extension elements, they would be ignored upon invokation of XSLT A (Thanks to molson for help on
this one)
Assorted bug fixes

Make some Ft.Lib.Uri functions meet the coding convention
Add fcore:normalize-uri
Add support for logger to extension elements
Bug fixes

regex fixes

more PathResolve fixes

PathResolve() was using urllib.basejoin() on OS paths, rather than URLs.

Changed output of file scheme in BaseJoin to be consistent with the
rest of the implementations.

Added UriException for error when resolving the URI

added support for repo URI scheme in BaseUriResolver,
 which FtssInputSource uses

implement Uri.PathResolve and Uri.BaseJoin
get rid of urllib.basejoin

New UriDict class (a UserDict) treats URIs that start with
'file://localhost/' and those that start with 'file:///' as
identical keys.

Added new _getScheme() helper to efficiently extract the scheme
of the given absolute URI.

OsPathToUri() returns 'file:///path/to/file' now rather than
'file:/path/to/file', and several bugs/inconsistencies in the
Windows version of urllib.pathname2url() have been worked around.

BaseUriResolver.normalize() now uses absolutize() instead of
urlparse.urljoin(). It always returns an absolute (scheme-having) URI
now, never just a path, but it still accepts a schemeless 'file' URI
as input.

UriToOsPath() needed to work around bugs in urllib.url2pathname()
and treat 'file://localhost/' same as 'file:///'

corrected inaccuracy in comments

raise UriExceptions in absolutize()

1. absolutize() added: it provides true RFC 2396 resolution of relative
URI references to absolute form. It is intended to be used in places
where we currently call urllib.basejoin() or urlparse.urljoin(). It has
not been optimized for speed nor has it been tested in any part of 4Suite.

2. _normalizeStrictPatternGroups() commented out (might be useful someday);

3. a few more notes about 'file' scheme added.

BaseUriResolver.normalize() was mishandling empty string URI refs

fixed URI reference regex so '\n' doesn't match

how'd that tab get in there

scheme determination in SchemeUriResolver was buggy

matchesUriRefSyntax() and matchesUriSyntax() were calling the wrong regex
init function

Moved comments about the regular expressions to saner locations.

Moved SPLIT_URIREF_PATTERN compilation to a separate function,
which greatly speeds up the first call to splitUriRef() or isAbsolute(),
since they no longer have to compile all the validation patterns.
This also eliminates Uche's FIXME.

BaseUriResolver.normalize() doc string and comment updates

Implement SchemeRegistryResolver
Bug fixes

Check in catalog support
Check in preliminary streamable XPointer support

Allows UriToOsPath to accept URIs with no schemes.

Add Ft.Lib.Uri.UriToOsPath
Fixes

Fixed so local files are opened binary

Work on the Lib test suite. Added ability to ignore funcions in coverage tests

matchesUriSyntax() was using wrong patterns;
added isAbsolute() convenience function;
minor doc edits

Fixed bug in absolute path resolution

patch from Alexander Smishlajev for splitUriRef() to better handle an
opaque part without a fragment

typo fix in InputSource comments;
Uri resolver normalize() updated to handle None or empty-string base URI;
test suite updates to generate proper base URIs for InputSource creation

Clean-ups to URI handling
grammer -> grammar
Support \" and \' in Versa strings
Pass through exceptions with Versa extensions
etc.

Fixes to Uri.OsPathToUri

Add Ft.Lib.Uri.OsPathToUri
Fix stylesheet include paths and add test
Pass through typeerrors in Versa function calls if they are not caused by the immediate invokation

Updated to compile regular expressions on demand

added IPv6 support to strict regexes, as per RFC 2732

replaced 'portable' versions of strict regexes with instructions on how to
 create them if you need them, and issues to be aware of

silly error

slight optimization in strict regexes ('port' group was redundant)

in splitUriRef(), needed to separate opaque part from fragment

Changed splitUriRef() to work with URI references that have an opaque part,
e.g. URNs

added splitUriRef()

corrected dup group name errors in strict regexes;
added _normalizeStrictPatternGroups function to clean up a match groupdict;
other minor fixes in strict regexes

added regexes & functions for lenient and strict URI validation;
added documentation strings to BaseUriResolver

made adjustments to imports in response to pychecker diagnosis of each .py
file in the source tree on 25 dec 2001. over 120 files are in this checkin.
the majority of the changes are removal of extraneous imports. in some
cases, missing imports were added, or the order of imports changed due to
circularity. a few minor fixes mostly related to error handling are
also a part of this update. a change to Ft/Lib/DistExt/pydoc.py is the
addition of a commented-out print in an exception handling block. uncomment
it to see import errors that xmldoc obscures.

cDomlette gets entity handling and XInclude
cDomlette becomes default wherever possible (e.g. need to subclass, need
for validation or need for mutation prevent)
Version -> 0.12.0a1

Generatation of documents now working

Moved many things

	Diffs between Use Text Field chime-windows-safe XPATH_OPTIMIZATIONS-branch XML1_0rc4 XML1_0rc3 XML1_0rc2 XML1_0b3 XML1_0b2 XML1_0_2 XML1_0_1 XML1_0-maint-merge XML1_0-maint T4SS-Drivers STREAMXPTR_prebranch STREAMXPTR_branch SERVER_REFACTORING-branch SERVER_OPTOMIZATIONS R1_0b6 R1_0b1 R1_0a4 R1_0a3 R1_0a1 R1_0 R0_12_0a3 R0_12_0a2 Pre-DAV-branch PSYCOPG-prebranch PSYCOPG-branch-premerge PSYCOPG-branch-merged PSYCOPG-branch PRE-DOMAIN-SCOPE-RENAME MX_DATETIME_PORT MAIN LOCAL_EXT_FUNCTIONS INPUT_SOURCE HEAD EXPAT-1-95-4-prebranch EXPAT-1-95-4-merged EXPAT-1-95-4-branch EXPAT-1-95-2-prebranch EXPAT-1-95-2-branch EASYINSTALL-prebranch EASYINSTALL-branch DUMP_OBJECTTYPE_ALLOWED-prebranch DUMP_OBJECTTYPE_ALLOWED-branch DRIVER_REFACTORING-prebranch DAV-2003-06 CDOMLETTE_2_GOLD and Use Text Field chime-windows-safe XPATH_OPTIMIZATIONS-branch XML1_0rc4 XML1_0rc3 XML1_0rc2 XML1_0b3 XML1_0b2 XML1_0_2 XML1_0_1 XML1_0-maint-merge XML1_0-maint T4SS-Drivers STREAMXPTR_prebranch STREAMXPTR_branch SERVER_REFACTORING-branch SERVER_OPTOMIZATIONS R1_0b6 R1_0b1 R1_0a4 R1_0a3 R1_0a1 R1_0 R0_12_0a3 R0_12_0a2 Pre-DAV-branch PSYCOPG-prebranch PSYCOPG-branch-premerge PSYCOPG-branch-merged PSYCOPG-branch PRE-DOMAIN-SCOPE-RENAME MX_DATETIME_PORT MAIN LOCAL_EXT_FUNCTIONS INPUT_SOURCE HEAD EXPAT-1-95-4-prebranch EXPAT-1-95-4-merged EXPAT-1-95-4-branch EXPAT-1-95-2-prebranch EXPAT-1-95-2-branch EASYINSTALL-prebranch EASYINSTALL-branch DUMP_OBJECTTYPE_ALLOWED-prebranch DUMP_OBJECTTYPE_ALLOWED-branch DRIVER_REFACTORING-prebranch DAV-2003-06 CDOMLETTE_2_GOLD
	Generate output suitable for use with a patch program
	Type of Diff should be a Colored Diff Long Colored Diff Unidiff Context Diff Side by Side

---

View only Branch: Show all branches XPATH_OPTIMIZATIONS-branch XML1_0-maint T4SS-Drivers STREAMXPTR_branch SERVER_REFACTORING-branch SERVER_OPTOMIZATIONS R0_12_0a3 PSYCOPG-branch MX_DATETIME_PORT MAIN INPUT_SOURCE EXPAT-1-95-4-branch EXPAT-1-95-2-branch EASYINSTALL-branch DUMP_OBJECTTYPE_ALLOWED-branch DAV-2003-06 CDOMLETTE_2_GOLD

---

Sort log by: Not sorted Commit date Revision

---

CVS administrator Powered by ViewCVS 1.0-dev

ViewCVS and CVS Help