[perl #130642] lex_allbrackets handling and S_tokereport

# New Ticket Created by  Hugo van der Sanden 
# Please include the string:  [perl #130642]
# in the subject line of all future correspondence about this issue. 
# <URL: https://rt.perl.org/Ticket/Display.html?id=130642 >


This is a bug report for perl from hv@crypt.org,
generated with the help of perlbug 1.40 running under perl 5.25.10.

-----------------------------------------------------------------
[Please describe your issue here]

I noticed that -DT fails to cope with certain tokens, tangled up with
the use of a bunch of magic numbers {1,2,4}<<24. I don't have much
understanding of this area, but I've pushed a branch hv/brackets as
a starting point.

If Zefram or anyone else can suggest improvements, I'm happy to make
them or hand it over. I think it particularly needs:
- better names for the macros and corresponding text for -DT;
- comments for each explaining their meaning and purpose;
- references to them (even if just in comments) if they are taken advantage
  of anywhere else.

I note in particular that nothing currently appears to set
TYPEFLAG_BRACKMINUS (at least, not using '<< 24'); I can't tell if that's
a bug or simply a placeholder.

Hugo

[Please do not change anything below this line]
-----------------------------------------------------------------
---
Flags:
    category=core
    severity=low
---
Site configuration information for perl 5.25.10:

Configured by hv at Mon Jan 23 20:29:32 GMT 2017.

Summary of my perl5 (revision 5 version 25 subversion 10) configuration:
  Commit id: e18c4116c82b2027a1e5d4e6b9a7214d60779053
  Platform:
    osname=linux
    osvers=3.13.0-101-generic
    archname=x86_64-linux
    uname='linux shad2 3.13.0-101-generic #148-ubuntu smp thu oct 20 22:08:32 utc 2016 x86_64 x86_64 x86_64 gnulinux '
    config_args='-des -Dcc=gcc -Dprefix=/opt/blead-d0 -Doptimize=-g -O0 -DDEBUGGING -Dusedevel -Uversiononly'
    hint=recommended
    useposix=true
    d_sigaction=define
    useithreads=undef
    usemultiplicity=undef
    use64bitint=define
    use64bitall=define
    uselongdouble=undef
    usemymalloc=n
    bincompat5005=undef
  Compiler:
    cc='gcc'
    ccflags ='-fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2'
    optimize='-g -O0'
    cppflags='-fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include'
    ccversion=''
    gccversion='4.8.4'
    gccosandvers=''
    intsize=4
    longsize=8
    ptrsize=8
    doublesize=8
    byteorder=12345678
    doublekind=3
    d_longlong=define
    longlongsize=8
    d_longdbl=define
    longdblsize=16
    longdblkind=3
    ivtype='long'
    ivsize=8
    nvtype='double'
    nvsize=8
    Off_t='off_t'
    lseeksize=8
    alignbytes=8
    prototype=define
  Linker and Libraries:
    ld='gcc'
    ldflags =' -fstack-protector -L/usr/local/lib'
    libpth=/usr/local/lib /usr/lib/gcc/x86_64-linux-gnu/4.8/include-fixed /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib /lib64 /usr/lib64
    libs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
    perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
    libc=libc-2.19.so
    so=so
    useshrplib=false
    libperl=libperl.a
    gnulibc_version='2.19'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs
    dlext=so
    d_dlsymun=undef
    ccdlflags='-Wl,-E'
    cccdlflags='-fPIC'
    lddlflags='-shared -g -O0 -L/usr/local/lib -fstack-protector'


---
@INC for perl 5.25.10:
    /opt/blead-d0/lib/perl5/site_perl/5.25.10/x86_64-linux
    /opt/blead-d0/lib/perl5/site_perl/5.25.10
    /opt/blead-d0/lib/perl5/5.25.10/x86_64-linux
    /opt/blead-d0/lib/perl5/5.25.10
    /opt/blead-d0/lib/perl5/site_perl/5.25.8
    /opt/blead-d0/lib/perl5/site_perl

---
Environment for perl 5.25.10:
    HOME=/home/hv
    LANG=C
    LANGUAGE=en_GB:en
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=<elided>
    PERL_BADLANG (unset)
    SHELL=/bin/bash

• [perl #130642] lex_allbrackets handling and S_tokereport by Hugo van der Sanden