# HG changeset patch # User Adam Chlipala # Date 1320519913 14400 # Node ID 1c9f8f06c1d6660f8192f80c49162c6e76a0aa22 # Parent 20f898c29525cdc811c2e7ec738c5136e0a6a0bb Support the full set of XHTML character entities diff -r 20f898c29525 -r 1c9f8f06c1d6 .hgignore --- a/.hgignore Sat Nov 05 13:12:07 2011 -0400 +++ b/.hgignore Sat Nov 05 15:05:13 2011 -0400 @@ -48,6 +48,9 @@ Makefile.coq *.vo +xml/parse +xml/entities.sml + syntax: regexp ^Makefile$ diff -r 20f898c29525 -r 1c9f8f06c1d6 Makefile.am --- a/Makefile.am Sat Nov 05 13:12:07 2011 -0400 +++ b/Makefile.am Sat Nov 05 15:05:13 2011 -0400 @@ -21,7 +21,7 @@ clean-local: rm -f src/*.mlton.grm.* src/*.mlton.lex.* \ - src/urweb.cm src/urweb.mlb + src/urweb.cm src/urweb.mlb xml/parse xml/entities.sml rm -rf .cm src/.cm src/urweb.cm: src/prefix.cm src/sources @@ -55,11 +55,18 @@ # MLTON += -profile $(PROFILE) #endif -bin/urweb: src/compiler.mlb src/urweb.mlb src/*.sig src/*.sml \ +bin/urweb: xml/entities.sml \ + src/compiler.mlb src/urweb.mlb src/*.sig src/*.sml \ src/urweb.mlton.lex.sml \ src/urweb.mlton.grm.sig src/urweb.mlton.grm.sml $(MLTON) -output $@ src/compiler.mlb +xml/entities.sml: xml/parse xml/xhtml-lat1.ent xml/xhtml-special.ent xml/xhtml-symbol.ent + xml/parse >xml/entities.sml + +xml/parse: xml/parse.sml + $(MLTON) xml/parse.sml + install-exec-emacs: if USE_EMACS mkdir -p $(DESTDIR)$(SITELISP) diff -r 20f898c29525 -r 1c9f8f06c1d6 Makefile.in --- a/Makefile.in Sat Nov 05 13:12:07 2011 -0400 +++ b/Makefile.in Sat Nov 05 15:05:13 2011 -0400 @@ -750,7 +750,7 @@ clean-local: rm -f src/*.mlton.grm.* src/*.mlton.lex.* \ - src/urweb.cm src/urweb.mlb + src/urweb.cm src/urweb.mlb xml/parse xml/entities.sml rm -rf .cm src/.cm src/urweb.cm: src/prefix.cm src/sources @@ -782,11 +782,18 @@ # MLTON += -profile $(PROFILE) #endif -bin/urweb: src/compiler.mlb src/urweb.mlb src/*.sig src/*.sml \ +bin/urweb: xml/entities.sml \ + src/compiler.mlb src/urweb.mlb src/*.sig src/*.sml \ src/urweb.mlton.lex.sml \ src/urweb.mlton.grm.sig src/urweb.mlton.grm.sml $(MLTON) -output $@ src/compiler.mlb +xml/entities.sml: xml/parse xml/xhtml-lat1.ent xml/xhtml-special.ent xml/xhtml-symbol.ent + xml/parse >xml/entities.sml + +xml/parse: xml/parse.sml + $(MLTON) xml/parse.sml + install-exec-emacs: @USE_EMACS_TRUE@ mkdir -p $(DESTDIR)$(SITELISP) @USE_EMACS_TRUE@ cp src/elisp/*.el $(DESTDIR)$(SITELISP)/ diff -r 20f898c29525 -r 1c9f8f06c1d6 src/sources --- a/src/sources Sat Nov 05 13:12:07 2011 -0400 +++ b/src/sources Sat Nov 05 15:05:13 2011 -0400 @@ -47,6 +47,11 @@ source.sml +utf8.sig +utf8.sml + +../xml/entities.sml + urweb.grm urweb.lex diff -r 20f898c29525 -r 1c9f8f06c1d6 src/urweb.lex --- a/src/urweb.lex Sat Nov 05 13:12:07 2011 -0400 +++ b/src/urweb.lex Sat Nov 05 15:05:13 2011 -0400 @@ -113,6 +113,14 @@ xmlString := false) +structure StringMap = BinaryMapFn(struct + type ord_key = string + val compare = String.compare + end) + +val entities = foldl (fn ((key, value), entities) => StringMap.insert (entities, key, value)) + StringMap.empty Entities.all + fun unescape loc s = let fun process (s, acc) = @@ -141,20 +149,16 @@ let val code = Substring.string (Substring.slice (code, 1, NONE)) in - Option.map chr (Int.fromString code) + Option.map Utf8.encode (Int.fromString code) end - else case Substring.string code of - "amp" => SOME #"&" - | "lt" => SOME #"<" - | "gt" => SOME #">" - | "quot" => SOME #"\"" - | _ => NONE + else + Option.map Utf8.encode (StringMap.find (entities, Substring.string code)) in case special of NONE => (ErrorMsg.errorAt' loc ("Unsupported XML character entity " ^ Substring.string code); "") - | SOME ch => process (s, Substring.full (String.str ch) :: pre :: acc) + | SOME sp => process (s, Substring.full sp :: pre :: acc) end end end diff -r 20f898c29525 -r 1c9f8f06c1d6 src/utf8.sig --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/src/utf8.sig Sat Nov 05 15:05:13 2011 -0400 @@ -0,0 +1,32 @@ +(* Copyright (c) 2011, Adam Chlipala + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * + * - Redistributions of source code must retain the above copyright notice, + * this list of conditions and the following disclaimer. + * - Redistributions in binary form must reproduce the above copyright notice, + * this list of conditions and the following disclaimer in the documentation + * and/or other materials provided with the distribution. + * - The names of contributors may not be used to endorse or promote products + * derived from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" + * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + *) + +(* UTF-8 conversion *) + +signature UTF8 = sig + val encode : int -> string +end diff -r 20f898c29525 -r 1c9f8f06c1d6 src/utf8.sml --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/src/utf8.sml Sat Nov 05 15:05:13 2011 -0400 @@ -0,0 +1,59 @@ +(* Copyright (c) 2011, Adam Chlipala + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * + * - Redistributions of source code must retain the above copyright notice, + * this list of conditions and the following disclaimer. + * - Redistributions in binary form must reproduce the above copyright notice, + * this list of conditions and the following disclaimer in the documentation + * and/or other materials provided with the distribution. + * - The names of contributors may not be used to endorse or promote products + * derived from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" + * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + *) + +(* UTF-8 conversion *) + +structure Utf8 :> UTF8 = struct + +fun byte n = str (chr (Word.toInt n)) + +fun encode n = + if n <= 0 then + raise Fail "Invalid character to UTF-8-encode" + else if n <= 0x7F then + str (chr n) + else if n <= 0x7FF then + let + val w = Word.fromInt n + val b1 = Word.orb (Word.fromInt (128 + 64), Word.>> (w, Word.fromInt 6)) + val b2 = Word.orb (Word.fromInt 128, Word.andb (w, Word.fromInt 63)) + in + byte b1 ^ byte b2 + end + else if n <= 0xFFFF then + let + val w = Word.fromInt n + val b1 = Word.orb (Word.fromInt (128 + 64 + 32), Word.>> (w, Word.fromInt 12)) + val b2 = Word.orb (Word.fromInt 128, Word.andb (Word.>> (w, Word.fromInt 6), Word.fromInt 63)) + val b3 = Word.orb (Word.fromInt 128, Word.andb (w, Word.fromInt 63)) + in + byte b1 ^ byte b2 ^ byte b3 + end + else + raise Fail "Exceeded supported range for UTF-8 characters" + +end diff -r 20f898c29525 -r 1c9f8f06c1d6 tests/entities.ur --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/tests/entities.ur Sat Nov 05 15:05:13 2011 -0400 @@ -0,0 +1,5 @@ +fun main () : transaction page = return + Hello world! & so on, © me today (8 €)
+ ♠ ♣ ♥ ♦
+ † DANGER † + diff -r 20f898c29525 -r 1c9f8f06c1d6 xml/parse.sml --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/xml/parse.sml Sat Nov 05 15:05:13 2011 -0400 @@ -0,0 +1,77 @@ +(* Copyright (c) 2011, Adam Chlipala + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * + * - Redistributions of source code must retain the above copyright notice, + * this list of conditions and the following disclaimer. + * - Redistributions in binary form must reproduce the above copyright notice, + * this list of conditions and the following disclaimer in the documentation + * and/or other materials provided with the distribution. + * - The names of contributors may not be used to endorse or promote products + * derived from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" + * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + *) + +(* Building SML code from XML entity tables *) + +fun main () = + let + fun doFile fname = + let + val inf = TextIO.openIn fname + + fun loop () = + case TextIO.inputLine inf of + NONE => TextIO.closeIn inf + | SOME line => + if String.isPrefix " Char.isSpace ch orelse ch = #">") line of + " + let + val exp = if String.isPrefix "\"&#" exp andalso String.isSuffix ";\"" exp then + let + val middle = String.substring (exp, 3, size exp - 5) + in + if CharVector.all Char.isDigit middle then + middle + else if String.isPrefix "38;#" middle then + String.extract (middle, 4, NONE) + else + raise Fail "Bad entity expression [1]" + end + else + raise Fail "Bad entity expansion [2]" + in + print ("\t\t(\"" ^ ent ^ "\", " ^ exp ^ "),\n"); + loop () + end + | _ => raise Fail "Bad ENTITY line" + else + loop () + in + loop () + end + in + print "structure Entities = struct\n"; + print "\tval all = [\n"; + doFile "xml/xhtml-lat1.ent"; + doFile "xml/xhtml-special.ent"; + doFile "xml/xhtml-symbol.ent"; + print "\t(\"\", 0)]\n"; + print "end\n" + end + +val () = main () diff -r 20f898c29525 -r 1c9f8f06c1d6 xml/xhtml-lat1.ent --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/xml/xhtml-lat1.ent Sat Nov 05 15:05:13 2011 -0400 @@ -0,0 +1,196 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff -r 20f898c29525 -r 1c9f8f06c1d6 xml/xhtml-special.ent --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/xml/xhtml-special.ent Sat Nov 05 15:05:13 2011 -0400 @@ -0,0 +1,80 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff -r 20f898c29525 -r 1c9f8f06c1d6 xml/xhtml-symbol.ent --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/xml/xhtml-symbol.ent Sat Nov 05 15:05:13 2011 -0400 @@ -0,0 +1,237 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +