Wednesday, May 18, 2011

401 UNAUTHORIZED error when integrate Perl from Window server to SharePoint 2010 site through SOAP web service

We have engineers using perl on Linux and Unix running testing scripts. They would like to send the testing result directly from perl to SharePoint. The perl to SharePoint integration is critical for our engineers on their daily jobs.

Since we need to pass authentication, we are trying to pass this on window server first before moving to Linux or Unix. We have identified two good blogs on this topics and tried the approach.  However, we are getting 401 UNAUTHORIZED error against both NTLM and Kerberos SharePoint 2010 sites. We believe the NTLM perl package may not configured correctly or Perl may not connect to SharePoint Kerberos sites. Since we are not perl or SharePoint integration expertise, we could like to get some help.


We have tried on 2008 64bit window VM server before moving to Linux or Unix. Here are the steps we tried.
  1. Install ActivePerl 5.12.3
  2. Install SOAP::Lite using cpan command cpan -i SOAP:Lite since it’s difficult using ppm
  3. Setup SharePoint site and list and grant site collection admin to the test user account
  4. Tried the script from The SharePoint / Perl Connection (Part 1)
  5. Error The exception is on line: $lists = $soap->GetListCollection();
          401 Unauthorized at E:\Share\ws\ws.pl line 27
          Client-Warning: Unsupported authentication scheme 'ntlm'.

    6. I also tried to add the following line:
          use Authen::NTLM; ntlmv2(1);
          But got error
          E:\Share\ws>ws1.pl
          Can't locate Authen/NTLM.pm in @INC (@INC contains: C:/Perl64/site/lib C:/Perl64
/lib .) at E:\Share\ws\ws1.pl line 3.

It seems like the NTLM.pm is inside C:\Perl64\lib\LWP\Authen.

    7. I also tried to install LWP::Authen::NTLM from CPAN http://search.cpan.org/~gaas/libwww-perl/lib/LWP/Authen/Ntlm.pm. I’m not familiar the perl make installation.


Here is the script from John Wefler and I modified with our server and site.
================================================
print "Start\n";

use LWP::UserAgent;
use LWP::Debug;
use SOAP::Lite on_action => sub { "$_[0]$_[1]"; };
import SOAP::Data 'name', 'value';
sub SOAP::Transport::HTTP::Client::get_basic_credentials { return ('user' => 'password') };
our $sp_endpoint = 'http://sbx01/sites/Harry/_vti_bin/lists.asmx';
our $sp_domain = 'sbx01.na.qulacomm.com:80';
our $sp_username = 'adminaccount';
our $sp_password = 'password';

$debug = 1;
if ($debug) {
    LWP::Debug::level('+');
    SOAP::Lite->import(+trace => 'all');
}
my @ua_args = (keep_alive => 1);
my @credentials = ($sp_domain, "", $sp_username, $sp_password);
my $schema_ua = LWP::UserAgent->new(@ua_args);
$schema_ua->credentials(@credentials);
$soap = SOAP::Lite->proxy($sp_endpoint, @ua_args, credentials => \@credentials);
$soap->schema->useragent($schema_ua);
$soap->uri("http://schemas.microsoft.com/sharepoint/soap/");

$lists = $soap->GetListCollection();
quit(1, $lists->faultstring()) if defined $lists->fault();

sub lists_getid {
    my $title = shift;
    my @result = $lists->dataof('//GetListCollectionResult/Lists/List');
    foreach my $data (@result) {
        my $attr = $data->attr;
        return $attr->{ID}
        if ($attr->{Title} eq $title);
    }
    return undef;
}
sub lists_getitems
{
    my $listid = shift;
    my $in_listName = name('listName' => $listid);
    my $in_viewName = name('viewName' => '');
    my $in_rowLimit = name('rowLimit' => 99999);
    my $call = $soap->GetListItems($in_listName, $in_viewName, $in_rowLimit);
    quit(1, $call->faultstring()) if defined $call->fault();
    return $call->dataof('//GetListItemsResult/listitems/data/row');
}
my $list_id = lists_getid('Shared Documents');
print "List ID is: $list_id\n";
my @items = lists_getitems($list_id);
foreach my $data (@items) {
    my $attr = $data->attr;
    print Dumper($attr);
}

print "END\n";

=============================================




Here is the error I got:

I also listed some reference I got from web for your reference.
Since the final goal is to use perl on Unix and Linux, we found LWP does not have NTLM implemented on Unix/Linux through CPAN. However, both wget and curl  ( command line web clients ) support NTLM, and curl supports Kerberos. So we tried to use perl with wget to communicate with SharePoint.

Here is an example command from RHEL 5 command line that gets results back from the Lists web service:

wget --post-file /tmp/listrequest --header 'Content-Type: application/soap+xml; charset="utf-8"' --http-user='NA\mchiles' --http-password="putyourpwhere"  http://sharepoint.qualcomm.com/_vti_bin/Lists.asmx

note that I put the file for the request in /tmp/listrequest.

If you save this xml body to this file,  it should work
##


<?xml version="1.0" encoding="UTF-8"?><soap:Envelope xmlns:soap="http://www.w3.org/2003/05/soap-envelope"><soap:Body><GetListCollection xmlns="http://schemas.microsoft.com/sharepoint/soap/"/></soap:Body></soap:Envelope>

##

We are still try to get more SharePoint functions done through perl and please let me know if we make some progress.



    Friday, May 13, 2011

    Why Web Anlytical Reports "Custom Reports" always display empty reports?

    As we discussed in previous blog, Web Analytics service as part of Microsoft® SharePoint® Server 2010, is set of features to help you collect, report, and analyze the usage and effectiveness of your SharePoint Server 2010 deployment. Web Analytics features include reporting, Web Analytics workflow, and Web Analytics Web Part.

    As  SharePoint administrators, we could like to leverage the SharePoint 2010 Customized Web Analytic Reports to output weekly usage reports for management on the following metrics, total number of page views, average number of page views per day, total number of daily unique visitors and average number of unique visitors per day.  Some key benefits of the custom reports are the ability display visual charts and graphs in excel worksheets and the ability to aggregate report information into one consolidated report per web app and subsequently sent out to management.You could refer here for details on different reports.

    One issue we found in our environment is Custom Web Analytic Reports are not showing data. One of the consultant is able to generate the same Custom Web Analytic Reports with correct in his environment. We have tried everything from permissions to excel services without any luck. He are the steps to reproduce this error in our environments.

    1. Navigate to any site and select "Site Actions" -> "Site Setting" -> "Site Collection Web Analytics Reports" ->on the left navigation select "Top Browsers" ->"Analyze" tab. You will notice a valid report with data shown in the aspx page.

    2. Navigate to the ribbon group "Analyze" and choose "Customize Report". If you do not have this button, please refer the blog to enable this report.


    3. Name the report, "Harry.xlsx" note the document library and the report will be generated and displayed. You could navigate to the report under http://siteURL/AnalyticsReports/Forms/AllItems.aspx? and open in the browser. We found notice no data present in the report as shown in the below screen.

    We are not able to find any errors on the server and here is the log trace for your reference. It seems like call excel service succeed.

    User=NA\harryc    beb00800-6d2c-4fad-9f53-fe4ae1f8fc60

    Site=/sites/Harry    beb00800-6d2c-4fad-9f53-fe4ae1f8fc60


    WcfSendRequest: RemoteAddress: 'http://sbx01:32843/484f9d239d0046a2af1afd67257e47d4/ExcelService.asmx' Channel: 'Microsoft.Office.Excel.Server.CalculationServer.Proxy.IExcelServiceSoap' Action: 'http://schemas.microsoft.com/office/Excel/Server/WebServices/ExcelServerInternalService/ExcelServiceSoap/GetRange' MessageId: 'urn:uuid:6d0562b4-f247-47c7-b98c-b221c40cccb0'    beb00800-6d2c-4fad-9f53-fe4ae1f8fc60
     

    WcfReceiveRequest: LocalAddress: 'http://sbx01.na.qualcomm.com:32843/484f9d239d0046a2af1afd67257e47d4/ExcelService.asmx' Channel: 'System.ServiceModel.Channels.ServiceChannel' Action: 'http://schemas.microsoft.com/office/Excel/Server/WebServices/ExcelServerInternalService/ExcelServiceSoap/GetRange' MessageId: 'urn:uuid:6d0562b4-f247-47c7-b98c-b221c40cccb0'    beb00800-6d2c-4fad-9f53-fe4ae1f8fc60

    Entering monitored scope (ExecuteWcfServerOperation)    beb00800-6d2c-4fad-9f53-fe4ae1f8fc60

    ECS RequestId=2012    beb00800-6d2c-4fad-9f53-fe4ae1f8fc60


    User=0#.w|na\harryc    beb00800-6d2c-4fad-9f53-fe4ae1f8fc60


    Session=1.V21.5If3FmRT7OZcBtXSYrDI590.5.en-US5.en-US73.+0480#0000-11-00-01T02:00:00:0000#+0000#0000-03-00-02T02:00:00:0000#-006036.57a3d5b9-15eb-47cc-a8b4-aae9bb4a67941.N    beb00800-6d2c-4fad-9f53-fe4ae1f8fc60
     

    Document=http://sbx01/sites/Harry/AnalyticsReports/Harry.xlsx    beb00800-6d2c-4fad-9f53-fe4ae1f8fc60
     

    Result=Success    beb00800-6d2c-4fad-9f53-fe4ae1f8fc60
     

    Leaving Monitored Scope (ExecuteWcfServerOperation). Execution Time=4.05470527678797    beb00800-6d2c-4fad-9f53-fe4ae1f8fc60
     

    WcfReceiveRequest: LocalAddress: 'http://sbx01.na.qualcomm.com:32843/484f9d239d0046a2af1afd67257e47d4/ExcelService.asmx' Channel: 'System.ServiceModel.Channels.ServiceChannel' Action: 'http://schemas.microsoft.com/office/Excel/Server/WebServices/ExcelServerInternalService/ExcelServiceSoap/GetRange' MessageId: 'urn:uuid:6d0562b4-f247-47c7-b98c-b221c40cccb0'    beb00800-6d2c-4fad-9f53-fe4ae1f8fc60
     

    Entering monitored scope (ExecuteWcfServerOperation)    beb00800-6d2c-4fad-9f53-fe4ae1f8fc60
     

    Leaving Monitored Scope (ExecuteWcfServerOperation). Execution Time=0.222095266297812    beb00800-6d2c-4fad-9f53-fe4ae1f8fc60
     

    Result=Success    beb00800-6d2c-4fad-9f53-fe4ae1f8fc60

    If you have any solution on this, please share with us.

    Today as May 26, we had a good discussion with Microsoft DSE support and this issue has been resolved now. The trick is Web Anlytical Reports "Custom Reports" is based on excel service and it requires C2WT service in order to pass through the authentication. Here is the way to result the issues.

    Log in to central admin -> Central Adminstration -> System Settings -> Manage services on server -> start "Claims to Windows Token Service". See screen shot for your reference.


    You will see data generated for Web Anlytical Reports "Custom Reports" as below.


    At this time, you should be able to enable the Web Anlytical Reports "Custom Reports" and display the data!

    Wednesday, May 11, 2011

    Issue using Content Query Web Part to display audio or video files on SharePoint 2010

    As we described before there are several different ways you can play or video media file from a SharePoint 2010 site. There are prons and cons on each of the implementation. You could refer the blog for details.

    When we utilize the our of box Content Query Web Part to display audio or video files on SharePoint 2010,  we found one issue that is impacting the usability. The issue is when you have a Content Query Web Part on a page, it overlay and block any other system pages or configuration options. As a result, users  could not configure the web part or could not perform other activities such as create list since those options are overlapped blocked by the black box of the Content Query Web Part. Here are some examples of the issue.

    1. If you try to configure the Content Query Web Part, the menu is blocked by the web part and it's very difficult to identify the Configure Web part menu option. See the screen shot.



    2. If you want to create other lists or pages from "Site Actions" -> "More Options", the whole Create page might be blocked by the Content Query Web Part.


    Of cause, we could use different ways to play or video media file from a SharePoint 2010 site. However, some users just want to get this working to display YouTube videos as I setup even back in SharePoint 2007 version. At this time, I'm not able to resolve this issue.

    Does anyone have similar issue and have any solution or workaround?

    Play video and media files in SharePoint 2010 sites part I – Option comparison

    There are several different ways you can play or video media file from a SharePoint 2010 site. In summary, we have explored three different ways on SharePoint 2010. Here is the list of three different ways we have explored on SharePoint 2010.
    1. Use out of box web part to play audio or video media file
    2. Use SharePoint 2010 to play audio or video media files on Media server
    3. Use third-party media player to play audio or video media file
    There are pros and cons for each different one that I would like to share with you. Based on your business requirement, you may select different implementation. Let's start with out of box web parts.

    1. Use out of box web part to play audio or video media file
    There are three different ways you could use out of box web part to play media file.You could check this excellent blog how to setup each of them in details.
    • You can add a Media Web Part to the page to play an individual audio or video file.
    • You can configure dynamic displays of video or audio by configuring the Content Query Web Part to display audio or video files
    • Or you can embed a video from a third-party video sharing site by using embed code.

    The first and easiest way is to use the Media Web Part to play media file. The pros is it's easy to setup and you could follow up this excellent blog on the procedure. However,  the cons is if you use embed code or sharing links for pages that host video, they will not work. For example, you could not play youtube video using youtube URL like http://www.youtube.com/v/09-M-S7Og0o&hl=en_US&fs=1& in the configuration. This approach only display the static media file and could not display dynamic file. There are limited formats this web part will support

    The second option to use Content Editor Web Part to play media file. The pros for this is you could display one file or multiple files dynamically from the filter. The cons is that you need to know little xml file format. Here is example to setup to play youtube.
    • Use Content Editor Web partEdit page and add web part
    • Select “Content Editor Web Part” under “Media and Content” and add to the page
    • Click the empty space of the web part and it will display the ribbons
    • Under Editing Tools, click HTML and then Edit HTML Source and type in the following code
                   <div align="center">
                           <embed src=" http://www.youtube.com/v/7ByiLIosTPA&hl=en_US&fs=1&"
                                    autoplay="false"
                                    loop="false"
                                    width="600"
                                    height="400"
                           </embed>
                  </div>


    You could also add JQuery to display list of the files for this web part. Please see other blogs for details.

    The third option to use embed code from a third-party video-sharing website to add a video to a page, you can do so by adding the embed code to the Content Editor Web Part. The pros of this approach is very flexible and the cons is you need Designer permissions to do this and may be very tedious for non programmers. The code is similar like this.


                 <object width="480" height="385">
                        <param name="movie" value="http://www.youtube.com/v/EODwh5hS7ZY&hl=en_GB&fs=1&rel=0&color1=0x2b405b&color2=0x6b8ab6"></param>
                       <param name="allowFullScreen" value="true"></param>
                       <param name="allowscriptaccess" value="always"></param>
                       <embed src="http://www.youtube.com/v/EODwh5hS7ZY&hl=en_GB&fs=1&rel=0&color1=0x2b405b&color2=0x6b8ab6" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" width="480" height="385"></embed>
                   </object>

    2. Use SharePoint 2010 to play audio or video media files on Media server
    Microsoft provides Streaming Media Services that enables you to manage delivery of live or on-demand digital media content, including Windows Media Audio (WMA) and Windows Media Video (WMV), over networks by using administrative interfaces in Microsoft Windows Media Services 2008. If you need to check the details on the installation, you could check the SharePoint Gorge blog.

    The pros of this approach is you have the separate Media Server to handle all the different format of the media and the size of the file. As a result, it could support all different media formats and large files. The cons is the installation is complicated especially on the Video Transcoding installation and configuration. In our case, we have setup the architecture to allow SharePoint users to download files to SharePoint and we handle the media file transcode and decode from the back end. Here is the summary of the architecture for your reference.

    Here are the procedure we handle the media files so end users will not even know media files are steaming from media server.
    • A user uploads a video to the Video Library
    • The event receiver copies the video and a configuration file to a share on the WMS server
    • Windows service on the WMS server sees the new configuration file, reads the configuration information and uses that to transcode the video if necessary (WMV files do not need to be transcoded)
    • After transcoding the video, the Windows service will place the final video into the folder mapped to the WMS Publishing Point (making it available for on-demand streaming)
    As I mentioned before, the major problem is the codec installation on the Media Server in order to support multiple formats. The sequence of the codec installation is also critical to make sure all formats could be played. Here is the partial codex we installed with proper sequence.
    •      Install Cook codec            
    •      Install FLV Splitter
    •      Install RRM Splitter
    •      Install Real Alternative
    •      Install MpegDecoder012
    Some of the codec are free to download but several are not. If you need to support some formats, you need to purchase the codec.

    3. Use third-party media player to play
    After we explore the out of box web parts and Microsoft Media Server, we found neither of them could satisfy our end users requirement that is users should be able to setup the page and play different media formats themselves without IT involvement. Out of box web parts could not support all media formats or need some coding skills. Microsoft Media Server needs IT involvement to setup the back end and configure the connections to Mediaa server. If you want to have a simple way to allow end users to utilize SharePoint 2010 to play media and video, the following third party Media Player Rotator Web Part probably is one of the best options for you now.

    We looked at this Media Player Rotator Web Part. End users could either selects a random media file, a specific media file or a media playlist from the specified Sharepoint Picture or Document Library. The Microsoft Media Player plug-in is used to play back the media files.

    Supported file formats are MP3, SWF (Flash), FLV/F4V (Flash Video), PDF, AVI, WMV, MP4, MPG, MPEG, YouTube, ASX, RTMP (Streaming Media), MOV and M4V (QuickTime). You could display
    a single static movie or list of files. You could display YouTube videos or any files from Sharepoint List or Library.

    There are two drawbacks on this web parts. It does not support Target Audience as out of Media Web part does. It does not support streaming and as a result, web part requires media players (FLV, Windows media and Quick time) should be installed on client or server systems, otherwise it won’t play any video file.

    With more demanding on the Media and video file support from SharePoint users, I'm hoping Microsoft could come up much easier way to support all different formats in the near future. May be SharePoint 2015?

    Tuesday, May 10, 2011

    How to fix SharePoint 2010 excel service error "System.IO.FileLoadException: Loading this assembly would produce a different grant set from other instances"?

    If you enable to display excel file in browser on SahrePoint 2010, you could setup using the following steps.

    Go to list library such as Shared Document -> Library -> Library Settings->Advanced settings->Opening Documents in the Browser -> select "Open in the browser".

    However, we run into error listed bellow:

    ExcelService.PostProcessRequest: Excel Server Exception: System.TypeInitializationException: The type initializer for 'Microsoft.Office.Excel.Server.CalculationServer.ExcelServerApp' threw an exception. ---> System.IO.FileLoadException: Loading this assembly would produce a different grant set from other instances. (Exception from HRESULT: 0x80131401)     at Microsoft.Office.Server.Diagnostics.FirstChanceHandler.ExceptionFilter(Boolean fRethrowException, TryBlock tryBlock, FilterBlock filter, CatchBlock catchBlock, FinallyBlock finallyBlock)     at Microsoft.Office.Server.Diagnostics.ULS.SendWatsonOnExceptionTag(UInt32 tagID, ULSCatBase categoryID, String output, Boolean fRethrowException, TryBlock tryBlock, CatchBlock catchBlock, FinallyBlock finallyBlock) .....

    In order to resolve this issue, you could try the following procedures.

    1. Verify Excel Services Application is running correctly from central admin
    2. Verify Excel Calculation Services is in Started status
    3.  Restart the Excel Calculation Services
    If Excel Calculation Services restart could resolve it, the worst case is you need to restart SharePoint 2010 servers and this issue should be resolve. I'm not sure the root cause of the issues right now. If you have better idea on this, please share with me.

    Thursday, May 5, 2011

    The security validation for this page is invalid error when enable SharePoint 2010 Site Recycle Bin features

    After we install the Microsoft IT SharePoint 2010 Site Recycle Bin previous named Site Delete Capture tool has been released from CodePlex. We had issue when creating our sites using customized site definition with Site Recycle Bin features activation. We noticed the below error in the log file, which saying the error while activating Site Recycle feature but the real issue is with customized feature after site created.

    “Failed to activate feature 'SiteRecycleBinSiteFeature' (Id 78b94e15-23ce-43f9-8036-babd847497d1) associated with site template 'QUALCOMMDEPT#1' at scope \"http://sbx01:80/sites/Harry1/ww/pp\".  Exception: Microsoft.SharePoint.SPException: The security validation for this page is invalid. Click Back in your Web browser, refresh the page, and try your operation again”


    The customized feature after site creation will setup groups w/ unique permissions, enabled auditing on Document library, add portal connection URL, add some content types, disable designer setting, and setup search setting.  The feature code looks like as following:

                               web.AllowUnsafeUpdates = true;
                                if (web.IsRootWeb)
                                {
                                    SPSite SiteCol = web.Site;
                                    CreateGroupPermisions(web);
                                    CreateAuditSettings(SiteCol);
                                    CreatePortalSettings(SiteCol);
                                    AddContentTypetoDocsLibs(web, "Shared Documents", "Link to a Document");                               
                                    CreateDesignerSettings(web.Site, false, false, false, false);
                                    CreateSearchSettings(web.Site);
                                }
                                web.Update();
                                web.AllowUnsafeUpdates = false;


    We found other developers have reported similar issues and here is the solution they recommend to resolve it.

    Add the following two lines besides AllowUnsafeUpdates = true.


                   //Disabling the from digest :-
                  SPWebapplication.FormDigestSettings.Enabled = false;

                  //Your original code listed above here
                  //Enabling the from digest :-
                  SPWebapplication.FormDigestSettings.Enabled = true;


    This way you should resolve the "The security validation for this page is invalid." SPException.

    Wednesday, May 4, 2011

    April 2011 CU for MOSS, SharePoint 2010, and Project Server 2010 published

    The April 2011 Cumulative Updates has been release for SharePoint Foundation 2010,  SharePoint 2010 and Project Server 2010. They are available for download at:
    Microsoft still hasn’t publish corresponding KB articles with descriptions of the CUs on the official site, so there may be more details soon. Please be careful on any CU and we had bad experience on all previous CU.If you upgrade project server, you also need to upgrade client components.

    I’ve already noticed someone publish one issue that library not available after April CU upgrade. Since Service Pack 1 for SharePoint is right around the corner. We may exam the release note before jump to this CU upgrade this time as Dan Holme described. In addition, there are many features will be introduced in SP1. See blog for details.